Cached Functions and Methods

Cached Functions and Methods

AUTHORS:

  • William Stein: initial version, (inspired by conversation with Justin Walker)
  • Mike Hansen: added doctests and made it work with class methods.
  • Willem Jan Palenstijn: add CachedMethodCaller for binding cached methods to instances.
  • Tom Boothby: added DiskCachedFunction.
  • Simon King: improved performance, more doctests, cython version, CachedMethodCallerNoArgs, weak cached function, cached special methods.
  • Julian Rueth (2014-03-19, 2014-05-09): added key parameter, allow caching for unhashable elements

EXAMPLES:

By trac ticket #11115, cached functions and methods are now also available in Cython code. The following examples cover various ways of usage.

Python functions:

sage: @cached_function
....: def test_pfunc(x):
....:     '''
....:     Some documentation
....:     '''
....:     return -x
sage: test_pfunc(5) is test_pfunc(5)
True

In some cases, one would only want to keep the result in cache as long as there is any other reference to the result. By trac ticket #12215, this is enabled for UniqueRepresentation, which is used to create unique parents: If an algebraic structure, such as a finite field, is only temporarily used, then it will not stay in cache forever. That behaviour is implemented using weak_cached_function, that behaves the same as cached_function, except that it uses a WeakValueDictionary for storing the results.

sage: from sage.misc.cachefunc import weak_cached_function
sage: class A: pass
sage: @weak_cached_function
....: def f():
....:     print "doing a computation"
....:     return A()
sage: a = f()
doing a computation

The result is cached:

sage: b = f()
sage: a is b
True

However, if there are no strong references left, the result may be garbage collected, and thus a new computation would take place:

sage: del a
sage: del b
sage: import gc
sage: n = gc.collect()
sage: a = f()
doing a computation

Cython cdef functions do not allow arbitrary decorators. However, one can wrap a Cython function and turn it into a cached function, by trac ticket #11115. We need to provide the name that the wrapped method or function should have, since otherwise the name of the original function would be used:

sage: cython('''cpdef test_funct(x): return -x''')
sage: wrapped_funct = cached_function(test_funct, name='wrapped_funct')
sage: wrapped_funct
Cached version of <built-in function test_funct>
sage: wrapped_funct.__name__
'wrapped_funct'
sage: wrapped_funct(5)
-5
sage: wrapped_funct(5) is wrapped_funct(5)
True

We can proceed similarly for cached methods of Cython classes, provided that they allow attribute assignment or have a public attribute __cached_methods of type <dict>. Since trac ticket #11115, this is the case for all classes inheriting from Parent. See below for a more explicit example. By trac ticket #12951, cached methods of extension classes can be defined by simply using the decorater. However, an indirect approach is still needed for cpdef methods:

sage: cython_code = ['cpdef test_meth(self,x):',
....: '    "some doc for a wrapped cython method"',
....: '    return -x',
....: 'from sage.all import cached_method',
....: 'from sage.structure.parent cimport Parent',
....: 'cdef class MyClass(Parent):',
....: '    @cached_method',
....: '    def direct_method(self, x):',
....: '        "Some doc for direct method"',
....: '        return 2*x',
....: '    wrapped_method = cached_method(test_meth,name="wrapped_method")']
sage: cython(os.linesep.join(cython_code))
sage: O = MyClass()
sage: O.direct_method
Cached version of <method 'direct_method' of '...MyClass' objects>
sage: O.wrapped_method
Cached version of <built-in function test_meth>
sage: O.wrapped_method.__name__
'wrapped_method'
sage: O.wrapped_method(5)
-5
sage: O.wrapped_method(5) is O.wrapped_method(5)
True
sage: O.direct_method(5)
10
sage: O.direct_method(5) is O.direct_method(5)
True

In some cases, one would only want to keep the result in cache as long as there is any other reference to the result. By trac ticket #12215, this is enabled for UniqueRepresentation, which is used to create unique parents: If an algebraic structure, such as a finite field, is only temporarily used, then it will not stay in cache forever. That behaviour is implemented using weak_cached_function, that behaves the same as cached_function, except that it uses a WeakValueDictionary for storing the results.

sage: from sage.misc.cachefunc import weak_cached_function
sage: class A: pass
sage: @weak_cached_function
....: def f():
....:     print "doing a computation"
....:     return A()
sage: a = f()
doing a computation

The result is cached:

sage: b = f()
sage: a is b
True

However, if there are no strong references left, the result may be garbage collected, and thus a new computation would take place:

sage: del a
sage: del b
sage: import gc
sage: n = gc.collect()
sage: a = f()
doing a computation

By trac ticket #11115, even if a parent does not allow attribute assignment, it can inherit a cached method from the parent class of a category (previously, the cache would have been broken):

sage: cython_code = ["from sage.all import cached_method, cached_in_parent_method, Category, Objects",
....: "class MyCategory(Category):",
....: "    @cached_method",
....: "    def super_categories(self):",
....: "        return [Objects()]",
....: "    class ElementMethods:",
....: "        @cached_method",
....: "        def element_cache_test(self):",
....: "            return -self",
....: "        @cached_in_parent_method",
....: "        def element_via_parent_test(self):",
....: "            return -self",
....: "    class ParentMethods:",
....: "        @cached_method",
....: "        def one(self):",
....: "            return self.element_class(self,1)",
....: "        @cached_method",
....: "        def invert(self, x):",
....: "            return -x"]
sage: cython('\n'.join(cython_code))
sage: C = MyCategory()

In order to keep the memory footprint of elements small, it was decided to not support the same freedom of using cached methods for elements: If an instance of a class derived from Element does not allow attribute assignment, then a cached method inherited from the category of its parent will break, as in the class MyBrokenElement below.

However, there is a class ElementWithCachedMethod that has generally a slower attribute access, but fully supports cached methods. We remark, however, that cached methods are much faster if attribute access works. So, we expect that ElementWithCachedMethod will hardly by used.

sage: cython_code = ["from sage.structure.element cimport Element, ElementWithCachedMethod",
....: "cdef class MyBrokenElement(Element):",
....: "    cdef public object x",
....: "    def __init__(self,P,x):",
....: "        self.x=x",
....: "        Element.__init__(self,P)",
....: "    def __neg__(self):",
....: "        return MyBrokenElement(self.parent(),-self.x)",
....: "    def _repr_(self):",
....: "        return '<%s>'%self.x",
....: "    def __hash__(self):",
....: "        return hash(self.x)",
....: "    def __cmp__(left, right):",
....: "        return (<Element>left)._cmp(right)",
....: "    def __richcmp__(left, right, op):",
....: "        return (<Element>left)._richcmp(right,op)",
....: "    cdef int _cmp_c_impl(left, Element right) except -2:",
....: "        return cmp(left.x,right.x)",
....: "    def raw_test(self):",
....: "        return -self",
....: "cdef class MyElement(ElementWithCachedMethod):",
....: "    cdef public object x",
....: "    def __init__(self,P,x):",
....: "        self.x=x",
....: "        ElementWithCachedMethod.__init__(self,P)",
....: "    def __neg__(self):",
....: "        return MyElement(self.parent(),-self.x)",
....: "    def _repr_(self):",
....: "        return '<%s>'%self.x",
....: "    def __hash__(self):",
....: "        return hash(self.x)",
....: "    def __cmp__(left, right):",
....: "        return (<Element>left)._cmp(right)",
....: "    def __richcmp__(left, right, op):",
....: "        return (<Element>left)._richcmp(right,op)",
....: "    cdef int _cmp_c_impl(left, Element right) except -2:",
....: "        return cmp(left.x,right.x)",
....: "    def raw_test(self):",
....: "        return -self",
....: "from sage.structure.parent cimport Parent",
....: "cdef class MyParent(Parent):",
....: "    Element = MyElement"]
sage: cython('\n'.join(cython_code))
sage: P = MyParent(category=C)
sage: ebroken = MyBrokenElement(P,5)
sage: e = MyElement(P,5)

The cached methods inherited by the parent works:

sage: P.one()
<1>
sage: P.one() is P.one()
True
sage: P.invert(e)
<-5>
sage: P.invert(e) is P.invert(e)
True

The cached methods inherited by MyElement works:

sage: e.element_cache_test()
<-5>
sage: e.element_cache_test() is e.element_cache_test()
True
sage: e.element_via_parent_test()
<-5>
sage: e.element_via_parent_test() is e.element_via_parent_test()
True

The other element class can only inherit a cached_in_parent_method, since the cache is stored in the parent. In fact, equal elements share the cache, even if they are of different types:

sage: e == ebroken
True
sage: type(e) == type(ebroken)
False
sage: ebroken.element_via_parent_test() is e.element_via_parent_test()
True

However, the cache of the other inherited method breaks, although the method as such works:

sage: ebroken.element_cache_test()
<-5>
sage: ebroken.element_cache_test() is ebroken.element_cache_test()
False

The cache can be emptied:

sage: a = test_pfunc(5)
sage: test_pfunc.clear_cache()
sage: a is test_pfunc(5)
False
sage: a = P.one()
sage: P.one.clear_cache()
sage: a is P.one()
False

Since e and ebroken share the cache, when we empty it for one element it is empty for the other as well:

sage: b = ebroken.element_via_parent_test()
sage: e.element_via_parent_test.clear_cache()
sage: b is ebroken.element_via_parent_test()
False

Introspection works:

sage: from sage.misc.edit_module import file_and_line
sage: from sage.misc.sageinspect import sage_getdoc, sage_getfile, sage_getsource
sage: print sage_getdoc(test_pfunc)
   Some documentation
sage: print sage_getdoc(O.wrapped_method)
some doc for a wrapped cython method

sage: print sage_getdoc(O.direct_method)
Some doc for direct method

sage: print sage_getsource(O.wrapped_method)
cpdef test_meth(self,x):
    "some doc for a wrapped cython method"
    return -x
sage: print sage_getsource(O.direct_method)
def direct_method(self, x):
    "Some doc for direct method"
    return 2*x

It is a very common special case to cache a method that has no arguments. In that special case, the time needed to access the cache can be drastically reduced by using a special implementation. The cached method decorator automatically determines which implementation ought to be chosen. A typical example is sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal.gens() (no arguments) versus sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal.groebner_basis() (several arguments):

sage: P.<a,b,c,d> = QQ[]
sage: I = P*[a,b]
sage: I.gens()
[a, b]
sage: I.gens() is I.gens()
True
sage: I.groebner_basis()
[a, b]
sage: I.groebner_basis() is I.groebner_basis()
True
sage: type(I.gens)
<type 'sage.misc.cachefunc.CachedMethodCallerNoArgs'>
sage: type(I.groebner_basis)
<type 'sage.misc.cachefunc.CachedMethodCaller'>

By trac ticket #12951, the cached_method decorator is also supported on non-c(p)def methods of extension classes, as long as they either support attribute assignment or have a public attribute of type <dict> called __cached_methods. The latter is easy:

sage: cython_code = [
....: "from sage.misc.cachefunc import cached_method",
....: "cdef class MyClass:",
....: "    cdef public dict __cached_methods",
....: "    @cached_method",
....: "    def f(self, a,b):",
....: "        return a*b"]
sage: cython(os.linesep.join(cython_code))
sage: P = MyClass()
sage: P.f(2,3)
6
sage: P.f(2,3) is P.f(2,3)
True

Providing attribute access is a bit more tricky, since it is needed that an attribute inherited by the instance from its class can be overridden on the instance. That is why providing a __getattr__ would not be enough in the following example:

sage: cython_code = [
....: "from sage.misc.cachefunc import cached_method",
....: "cdef class MyOtherClass:",
....: "    cdef dict D",
....: "    def __init__(self):",
....: "        self.D = {}",
....: "    def __setattr__(self, n,v):",
....: "        self.D[n] = v",
....: "    def __getattribute__(self, n):",
....: "        try:",
....: "            return self.D[n]",
....: "        except KeyError:",
....: "            pass",
....: "        return getattr(type(self),n).__get__(self)",
....: "    @cached_method",
....: "    def f(self, a,b):",
....: "        return a+b"]
sage: cython(os.linesep.join(cython_code))
sage: Q = MyOtherClass()
sage: Q.f(2,3)
5
sage: Q.f(2,3) is Q.f(2,3)
True

Note that supporting attribute access is somehow faster than the easier method:

sage: timeit("a = P.f(2,3)")   # random
625 loops, best of 3: 1.3 µs per loop
sage: timeit("a = Q.f(2,3)")   # random
625 loops, best of 3: 931 ns per loop

Some immutable objects (such as \(p\)-adic numbers) cannot implement a reasonable hash function because their == operator has been modified to return True for objects which might behave differently in some computations:

sage: K.<a> = Qq(9)
sage: b = a.add_bigoh(1)
sage: c = a + 3
sage: b
a + O(3)
sage: c
a + 3 + O(3^20)
sage: b == c
True
sage: b == a
True
sage: c == a
False

If such objects defined a non-trivial hash function, this would break caching in many places. However, such objects should still be usable in caches. This can be achieved by defining an appropriate method _cache_key:

sage: hash(b)
Traceback (most recent call last):
...
TypeError: unhashable type: 'sage.rings.padics.padic_ZZ_pX_CR_element.pAdicZZpXCRElement'
sage: @cached_method
....: def f(x): return x == a
sage: f(b)
True
sage: f(c) # if b and c were hashable, this would return True
False

sage: b._cache_key()
(..., ((0, 1),), 0, 1)
sage: c._cache_key()
(..., ((0, 1), (1,)), 0, 20)

Note

This attribute will only be accessed if the object itself is not hashable.

An implementation must make sure that for elements a and b, if a != b, then also a._cache_key() != b._cache_key(). In practice this means that the _cache_key should always include the parent as its first argument:

sage: S.<a> = Qq(4)
sage: d = a.add_bigoh(1)
sage: b._cache_key() == d._cache_key() # this would be True if the parents were not included
False
class sage.misc.cachefunc.CachedFunction

Bases: object

Create a cached version of a function, which only recomputes values it hasn’t already computed. Synonyme: cached_function

INPUT:

  • f – a function
  • name – (optional string) name that the cached version of f should be provided with
  • key – (optional callable) takes the input and returns a key for the cache, typically one would use this to normalize input

If f is a function, do either g = CachedFunction(f) or g = cached_function(f) to make a cached version of f, or put @cached_function right before the definition of f (i.e., use Python decorators):

@cached_function
def f(...):
    ....

The inputs to the function must be hashable or they must define sage.structure.sage_object.SageObject._cache_key().

EXAMPLES:

sage: @cached_function
....: def mul(x, y=2):
....:     return x*y
sage: mul(3)
6

We demonstrate that the result is cached, and that, moreover, the cache takes into account the various ways of providing default arguments:

sage: mul(3) is mul(3,2)
True
sage: mul(3,y=2) is mul(3,2)
True

The user can clear the cache:

sage: a = mul(4)
sage: mul.clear_cache()
sage: a is mul(4)
False

It is also possible to explicitly override the cache with a different value:

sage: mul.set_cache('foo',5)
sage: mul(5,2)
'foo'

The parameter key can be used to ignore parameters for caching. In this example we ignore the parameter algorithm:

sage: @cached_function(key=lambda x,y,algorithm: (x,y))
....: def mul(x, y, algorithm="default"):
....:     return x*y
sage: mul(1,1,algorithm="default") is mul(1,1,algorithm="algorithm") is mul(1,1) is mul(1,1,'default')
True
cache
clear_cache()

Clear the cache dictionary.

EXAMPLES:

sage: g = CachedFunction(number_of_partitions)
sage: a = g(5)
sage: g.get_cache()
{((5, 'default'), ()): 7}
sage: g.clear_cache()
sage: g.get_cache()
{}
f
get_cache()

Returns the cache dictionary.

EXAMPLES:

sage: g = CachedFunction(number_of_partitions)
sage: a = g(5)
sage: g.get_cache()
{((5, 'default'), ()): 7}
get_key(*args, **kwds)

Return the key in the cache to be used when args and kwds are passed in as parameters.

EXAMPLES:

sage: @cached_function
....: def foo(x):
....:     return x^2
sage: foo(2)
4
sage: foo.get_key(2)
((2,), ())
sage: foo.get_key(x=3)
((3,), ())
is_in_cache(*args, **kwds)

Checks if the argument list is in the cache.

EXAMPLES:

sage: class Foo:
....:     def __init__(self, x):
....:         self._x = x
....:     @cached_method
....:     def f(self, z, y=0):
....:         return self._x*z+y
sage: a = Foo(2)
sage: a.f.is_in_cache(3)
False
sage: a.f(3)
6
sage: a.f.is_in_cache(3,y=0)
True

TESTS:

Check that trac ticket #16316 has been fixed, i.e., caching works for immutable unhashable objects which define sage.structure.sage_object.SageObject._cache_key():

sage: @cached_function
....: def f(x): return x
sage: K.<u> = Qq(4)
sage: x = K(1,1); x
1 + O(2)
sage: f.is_in_cache(x)
False
sage: f(x)
1 + O(2)
sage: f.is_in_cache(x)
True
precompute(arglist, num_processes=1)

Cache values for a number of inputs. Do the computation in parallel, and only bother to compute values that we haven’t already cached.

EXAMPLES:

sage: @cached_function
....: def oddprime_factors(n):
....:     l = [p for p,e in factor(n) if p != 2]
....:     return len(l)
sage: oddprime_factors.precompute(range(1,100), 4)
sage: oddprime_factors.cache[(25,),()]
1
set_cache(value, *args, **kwds)

Set the value for those args and keyword args Mind the unintuitive syntax (value first). Any idea on how to improve that welcome!

EXAMPLES:

sage: g = CachedFunction(number_of_partitions)
sage: a = g(5)
sage: g.get_cache()
{((5, 'default'), ()): 7}
sage: g.set_cache(17, 5)
sage: g.get_cache()
{((5, 'default'), ()): 17}
sage: g(5)
17

TESTS:

Check that trac ticket #16316 has been fixed, i.e., caching works for immutable unhashable objects which define sage.structure.sage_object.SageObject._cache_key():

sage: @cached_function
....: def f(x): return x
sage: K.<u> = Qq(4)
sage: x = K(1,1); x
1 + O(2)
sage: f.set_cache(x,x)
sage: f.is_in_cache(x)
True

DEVELOPER NOTE:

Is there a way to use the following intuitive syntax?

sage: g(5) = 19    # todo: not implemented
sage: g(5)         # todo: not implemented
19
class sage.misc.cachefunc.CachedInParentMethod

Bases: sage.misc.cachefunc.CachedMethod

A decorator that creates a cached version of an instance method of a class.

In contrast to CachedMethod, the cache dictionary is an attribute of the parent of the instance to which the method belongs.

ASSUMPTION:

This way of caching works only if

NOTE:

For proper behavior, the method must be a pure function (no side effects). If this decorator is used on a method, it will have identical output on equal elements. This is since the element is part of the hash key. Arguments to the method must be hashable or define sage.structure.sage_object.SageObject._cache_key(). The instance it is assigned to must be hashable.

Examples can be found at cachefunc.

class sage.misc.cachefunc.CachedMethod

Bases: object

A decorator that creates a cached version of an instance method of a class.

Note

For proper behavior, the method must be a pure function (no side effects). Arguments to the method must be hashable or transformed into something hashable using key or they must define sage.structure.sage_object.SageObject._cache_key().

EXAMPLES:

sage: class Foo(object):
....:     @cached_method
....:     def f(self, t, x=2):
....:         print 'computing'
....:         return t**x
sage: a = Foo()

The example shows that the actual computation takes place only once, and that the result is identical for equivalent input:

sage: res = a.f(3, 2); res
computing
9
sage: a.f(t = 3, x = 2) is res
True
sage: a.f(3) is res
True

Note, however, that the CachedMethod is replaced by a CachedMethodCaller or CachedMethodCallerNoArgs as soon as it is bound to an instance or class:

sage: P.<a,b,c,d> = QQ[]
sage: I = P*[a,b]
sage: type(I.__class__.gens)
<type 'sage.misc.cachefunc.CachedMethodCallerNoArgs'>

So, you would hardly ever see an instance of this class alive.

The parameter key can be used to pass a function which creates a custom cache key for inputs. In the following example, this parameter is used to ignore the algorithm keyword for caching:

sage: class A(object):
....:     def _f_normalize(self, x, algorithm): return x
....:     @cached_method(key=_f_normalize)
....:     def f(self, x, algorithm='default'): return x
sage: a = A()
sage: a.f(1, algorithm="default") is a.f(1) is a.f(1, algorithm="algorithm")
True
class sage.misc.cachefunc.CachedMethodCaller

Bases: sage.misc.cachefunc.CachedFunction

Utility class that is used by CachedMethod to bind a cached method to an instance.

Note

Since trac ticket #11115, there is a special implementation CachedMethodCallerNoArgs for methods that do not take arguments.

EXAMPLE:

sage: class A:
....:    @cached_method
....:    def bar(self,x):
....:        return x^2
sage: a = A()
sage: a.bar
Cached version of <function bar at 0x...>
sage: type(a.bar)
<type 'sage.misc.cachefunc.CachedMethodCaller'>
sage: a.bar(2) is a.bar(x=2)
True
get_key(*args, **kwds)

Convert arguments to the key for this instance’s cache.

EXAMPLES:

sage: class Foo:
....:     def __init__(self, x):
....:         self._x = x
....:     @cached_method
....:     def f(self, y, z=0):
....:         return self._x * y + z
sage: a = Foo(2)
sage: z = a.f(37)
sage: k = a.f.get_key(37); k
((37, 0), ())
sage: a.f.get_cache()[k] is z
True

Note that the method does not test whether there are too many arguments, or wrong argument names:

sage: a.f.get_key(1,2,3,x=4,y=5,z=6)
((1, 2, 3), (('x', 4), ('y', 5), ('z', 6)))

It does, however, take into account the different ways of providing named arguments, possibly with a default value:

sage: a.f.get_key(5)
((5, 0), ())
sage: a.f.get_key(y=5)
((5, 0), ())
sage: a.f.get_key(5,0)
((5, 0), ())
sage: a.f.get_key(5,z=0)
((5, 0), ())
sage: a.f.get_key(y=5,z=0)
((5, 0), ())
class sage.misc.cachefunc.CachedMethodCallerNoArgs

Bases: sage.misc.cachefunc.CachedFunction

Utility class that is used by CachedMethod to bind a cached method to an instance, in the case of a method that does not accept any arguments except self.

Note

The return value None would not be cached. So, if you have a method that does not accept arguments and may return None after a lengthy computation, then @cached_method should not be used.

EXAMPLE:

sage: P.<a,b,c,d> = QQ[]
sage: I = P*[a,b]
sage: I.gens
Cached version of <function gens at 0x...>
sage: type(I.gens)
<type 'sage.misc.cachefunc.CachedMethodCallerNoArgs'>
sage: I.gens is I.gens
True
sage: I.gens() is I.gens()
True

AUTHOR:

  • Simon King (2011-04)
clear_cache()

Clear the cache dictionary.

EXAMPLES:

sage: P.<a,b,c,d> = QQ[]
sage: I = P*[a,b]
sage: I.gens()
[a, b]
sage: I.gens.set_cache('bar')
sage: I.gens()
'bar'

The cache can be emptied and thus the original value will be reconstructed:

sage: I.gens.clear_cache()
sage: I.gens()
[a, b]
is_in_cache()

Answers whether the return value is already in the cache.

Note

Recall that a cached method without arguments can not cache the return value None.

EXAMPLE:

sage: P.<x,y> = QQ[]
sage: I = P*[x,y]
sage: I.gens.is_in_cache()
False
sage: I.gens()
[x, y]
sage: I.gens.is_in_cache()
True
set_cache(value)

Override the cache with a specific value.

Note

None is not suitable for a cached value. It would be interpreted as an empty cache, forcing a new computation.

EXAMPLES:

sage: P.<a,b,c,d> = QQ[]
sage: I = P*[a,b]
sage: I.gens()
[a, b]
sage: I.gens.set_cache('bar')
sage: I.gens()
'bar'

The cache can be emptied and thus the original value will be reconstructed:

sage: I.gens.clear_cache()
sage: I.gens()
[a, b]

The attempt to assign None to the cache fails:

sage: I.gens.set_cache(None)
sage: I.gens()
[a, b]
class sage.misc.cachefunc.CachedMethodPickle(inst, name, cache=None)

Bases: object

This class helps to unpickle cached methods.

Note

Since trac ticket #8611, a cached method is an attribute of the instance (provided that it has a __dict__). Hence, when pickling the instance, it would be attempted to pickle that attribute as well, but this is a problem, since functions can not be pickled, currently. Therefore, we replace the actual cached method by a place holder, that kills itself as soon as any attribute is requested. Then, the original cached attribute is reinstated. But the cached values are in fact saved.

EXAMPLES:

sage: R.<x, y, z> = PolynomialRing(QQ, 3)
sage: I = R*(x^3 + y^3 + z^3,x^4-y^4)
sage: I.groebner_basis()
[y^5*z^3 - 1/4*x^2*z^6 + 1/2*x*y*z^6 + 1/4*y^2*z^6,
 x^2*y*z^3 - x*y^2*z^3 + 2*y^3*z^3 + z^6,
 x*y^3 + y^4 + x*z^3, x^3 + y^3 + z^3]
sage: I.groebner_basis
Cached version of <function groebner_basis at 0x...>

We now pickle and unpickle the ideal. The cached method groebner_basis is replaced by a placeholder:

sage: J = loads(dumps(I))
sage: J.groebner_basis
Pickle of the cached method "groebner_basis"

But as soon as any other attribute is requested from the placeholder, it replaces itself by the cached method, and the entries of the cache are actually preserved:

sage: J.groebner_basis.is_in_cache()
True
sage: J.groebner_basis
Cached version of <function groebner_basis at 0x...>
sage: J.groebner_basis() == I.groebner_basis()
True

TESTS:

Since trac ticket #11115, there is a special implementation for cached methods that don’t take arguments:

sage: P.<a,b,c,d> = QQ[]
sage: I = P*[a,b]
sage: type(I.gens)
<type 'sage.misc.cachefunc.CachedMethodCallerNoArgs'>
sage: type(I.groebner_basis)
<type 'sage.misc.cachefunc.CachedMethodCaller'>

We demonstrate that both implementations can be pickled and preserve the cache. For that purpose, we assign nonsense to the cache. Of course, it is a very bad idea to override the cache in that way. So, please don’t try this at home:

sage: I.groebner_basis.set_cache('foo',algorithm='singular')
sage: I.groebner_basis(algorithm='singular')
'foo'
sage: I.gens.set_cache('bar')
sage: I.gens()
'bar'
sage: J = loads(dumps(I))
sage: J.gens()
'bar'
sage: J.groebner_basis(algorithm='singular')
'foo'

Anyway, the cache will be automatically reconstructed after clearing it:

sage: J.gens.clear_cache()
sage: J.gens()
[a, b]
sage: J.groebner_basis.clear_cache()
sage: J.groebner_basis(algorithm='singular')
[a, b]

AUTHOR:

  • Simon King (2011-01)
class sage.misc.cachefunc.CachedSpecialMethod

Bases: sage.misc.cachefunc.CachedMethod

Cached version of special python methods.

IMPLEMENTATION:

For new style classes C, it is not possible to override a special method, such as __hash__, in the __dict__ of an instance c of C, because Python will for efficiency reasons always use what is provided by the class, not by the instance.

By consequence, if __hash__ would be wrapped by using CachedMethod, then hash(c) will access C.__hash__ and bind it to c, which means that the __get__ method of CachedMethod will be called. But there, we assume that Python has already inspected __dict__, and thus a CachedMethodCaller will be created over and over again.

Here, the __get__ method will explicitly access the __dict__, so that hash(c) will rely on a single CachedMethodCaller stored in the __dict__.

EXAMPLES:

sage: class C:
....:     @cached_method
....:     def __hash__(self):
....:         print "compute hash"
....:         return int(5)
....:
sage: c = C()
sage: type(C.__hash__)
<type 'sage.misc.cachefunc.CachedMethodCallerNoArgs'>

The hash is computed only once, subsequent calls will use the value from the cache. This was implemented in trac ticket #12601.

sage: hash(c)       # indirect doctest
compute hash
5
sage: hash(c)
5
class sage.misc.cachefunc.ClearCacheOnPickle

Bases: object

This class implements an appropriate __getstate__ method that clears the cache of the methods (see @cached_method) before passing them on to the caller, typically the pickle and copy modules.

The implemented __getstate__ method calls the __getstate__ methods of classes later in the method resolution order. Therefore, classes which want this behaviour should inherit first from this one.

EXAMPLE:

In the following example, we create a Python class that inherits from multivariate polynomial ideals, but does not pickle cached values. We provide the definition in Cython, however, since interactive Cython definitions provide introspection by trac ticket #9976, whereas Python definitions don’t.

sage: P.<a,b,c,d> = QQ[]
sage: I = P*[a,b]
sage: classdef = ['from sage.misc.cachefunc import ClearCacheOnPickle',
....:    'from sage.all import QQ',
....:    'P = QQ["a","b","c","d"]; I = P*[P.gen(0),P.gen(1)]',
....:    'class MyClass(ClearCacheOnPickle,I.__class__):',
....:    '    def __init__(self,ring,gens):',
....:    '        I.__class__.__init__(self,ring,gens)',
....:    '    def __getnewargs__(self):',
....:    '        return (self._Ideal_generic__ring,self._Ideal_generic__gens)']
sage: cython('\n'.join(classdef))

We destroy the cache of two methods of I on purpose (demonstrating that the two different implementations of cached methods are correctly dealt with). Pickling I preserves the cache:

sage: I.gens.set_cache('bar')
sage: I.groebner_basis.set_cache('foo',algorithm='singular')
sage: J = loads(dumps(I))
sage: J.gens()
'bar'
sage: J.groebner_basis(algorithm='singular')
'foo'

However, if we have an ideal that additionally descends from ClearCacheOnPickle, the carefully corrupted cache is not pickled:

sage: A = MyClass(P,[a,b])
sage: A
Ideal (a, b) of Multivariate Polynomial Ring in a, b, c, d over Rational Field
sage: A.gens.set_cache('foo')
sage: A.groebner_basis.set_cache('bar',algorithm='singular')
sage: A.gens()
'foo'
sage: A.groebner_basis(algorithm='singular')
'bar'
sage: B = loads(dumps(A))
sage: B.gens()
[a, b]
sage: B.groebner_basis(algorithm='singular')
[a, b]
sage: A.gens()
'foo'
class sage.misc.cachefunc.DiskCachedFunction(f, dir, memory_cache=False, key=None)

Bases: sage.misc.cachefunc.CachedFunction

Works similar to CachedFunction, but instead, we keep the cache on disk (optionally, we keep it in memory too).

EXAMPLES:

sage: from sage.misc.cachefunc import DiskCachedFunction
sage: dir = tmp_dir()
sage: factor = DiskCachedFunction(factor, dir, memory_cache=True)
sage: f = factor(2775); f
3 * 5^2 * 37
sage: f is factor(2775)
True
class sage.misc.cachefunc.FileCache(dir, prefix='', memory_cache=False)

FileCache is a dictionary-like class which stores keys and values on disk. The keys take the form of a tuple (A,K)

  • A is a tuple of objects t where each t is an exact object which is uniquely identified by a short string.
  • K is a tuple of tuples (s,v) where s is a valid variable name and v is an exact object which is uniquely identified by a short string with letters [a-zA-Z0-9-._]

The primary use case is the DiskCachedFunction. If memory_cache == True, we maintain a cache of objects seen during this session in memory – but we don’t load them from disk until necessary. The keys and values are stored in a pair of files:

  • prefix-argstring.key.sobj contains the key only,
  • prefix-argstring.sobj contains the tuple (key,val)

where self[key] == val.

Note

We assume that each FileCache lives in its own directory. Use extreme caution if you wish to break that assumption.

file_list()

Return the list of files corresponding to self.

EXAMPLES:

sage: from sage.misc.cachefunc import FileCache
sage: dir = tmp_dir()
sage: FC = FileCache(dir, memory_cache = True, prefix='t')
sage: FC[((),())] = 1
sage: FC[((1,2),())] = 2
sage: FC[((1,),(('a',1),))] = 3
sage: for f in sorted(FC.file_list()): print f[len(dir):]
t-.key.sobj
t-.sobj
t-1_2.key.sobj
t-1_2.sobj
t-a-1.1.key.sobj
t-a-1.1.sobj
items()

Return a list of tuples (k,v) where self[k] = v.

EXAMPLES:

sage: from sage.misc.cachefunc import FileCache
sage: dir = tmp_dir()
sage: FC = FileCache(dir, memory_cache = False)
sage: FC[((),())] = 1
sage: FC[((1,2),())] = 2
sage: FC[((1,),(('a',1),))] = 3
sage: I = FC.items()
sage: I.sort(); print I
[(((), ()), 1), (((1,), (('a', 1),)), 3), (((1, 2), ()), 2)]
keys()

Return a list of keys k where self[k] is defined.

EXAMPLES:

sage: from sage.misc.cachefunc import FileCache
sage: dir = tmp_dir()
sage: FC = FileCache(dir, memory_cache = False)
sage: FC[((),())] = 1
sage: FC[((1,2),())] = 2
sage: FC[((1,),(('a',1),))] = 3
sage: K = FC.keys()
sage: K.sort(); print K
[((), ()), ((1,), (('a', 1),)), ((1, 2), ())]
values()

Return a list of values that are stored in self.

EXAMPLES:

sage: from sage.misc.cachefunc import FileCache
sage: dir = tmp_dir()
sage: FC = FileCache(dir, memory_cache = False)
sage: FC[((),())] = 1
sage: FC[((1,2),())] = 2
sage: FC[((1,),(('a',1),))] = 3
sage: FC[((),(('a',1),))] = 4
sage: v = FC.values()
sage: v.sort(); print v
[1, 2, 3, 4]
class sage.misc.cachefunc.WeakCachedFunction

Bases: sage.misc.cachefunc.CachedFunction

A version of CachedFunction using weak references on the values.

If f is a function, do either g = weak_cached_function(f) to make a cached version of f, or put @weak_cached_function right before the definition of f (i.e., use Python decorators):

@weak_cached_function
def f(...):
    ...

EXAMPLES:

sage: from sage.misc.cachefunc import weak_cached_function
sage: class A: pass
sage: @weak_cached_function
....: def f():
....:     print "doing a computation"
....:     return A()
sage: a = f()
doing a computation

The result is cached:

sage: b = f()
sage: a is b
True

However, if there are no strong references left, the result may be garbage collected, and thus a new computation would take place:

sage: del a
sage: del b
sage: import gc
sage: n = gc.collect()
sage: a = f()
doing a computation

The parameter key can be used to ignore parameters for caching. In this example we ignore the parameter algorithm:

sage: @weak_cached_function(key=lambda x,algorithm: x)
....: def mod_ring(x, algorithm="default"):
....:     return IntegerModRing(x)
sage: mod_ring(1,algorithm="default") is mod_ring(1,algorithm="algorithm") is mod_ring(1) is mod_ring(1,'default')
True
is_in_cache(*args, **kwds)

Check if the argument list is in the cache.

EXAMPLES:

sage: from sage.misc.cachefunc import weak_cached_function
sage: class A:
....:     def __init__(self, x):
....:         self.x = x
sage: @weak_cached_function
....: def f(n):
....:    return A(n)
sage: a = f(5)

The key 5 is in the cache, as long as there is a strong reference to the corresponding value:

sage: f.is_in_cache(5)
True

However, if there are no strong references left, the cached item is removed from cache after garbage collection:

sage: del a
sage: import gc
sage: n = gc.collect()
sage: f.is_in_cache(5)
False

TESTS:

Check that trac ticket #16316 has been fixed, i.e., caching works for immutable unhashable objects which define sage.structure.sage_object.SageObject._cache_key():

sage: from sage.misc.cachefunc import weak_cached_function
sage: @weak_cached_function
....: def f(x): return x
sage: K.<u> = Qq(4)
sage: R.<t> = K[]
sage: f.is_in_cache(t)
False
sage: f(t)
(1 + O(2^20))*t
sage: f.is_in_cache(t)
True
set_cache(value, *args, **kwds)

Set the value for those args and keyword args Mind the unintuitive syntax (value first). Any idea on how to improve that welcome!

It is required that the given value is weak referenceable. The item will be removed from cache if the value is garbage collected.

EXAMPLES:

sage: from sage.misc.cachefunc import weak_cached_function
sage: @weak_cached_function
....: def f(n):
....:     raise RuntimeError
sage: f.set_cache(ZZ, 5)
sage: f(5)
Integer Ring

TESTS:

Check that trac ticket #16316 has been fixed, i.e., caching works for immutable unhashable objects which define sage.structure.sage_object.SageObject._cache_key():

sage: from sage.misc.cachefunc import weak_cached_function
sage: @weak_cached_function
....: def f(x): return x
sage: K.<u> = Qq(4)
sage: R.<t> = K[]
sage: f.set_cache(t,t)
sage: f.is_in_cache(t)
True
sage.misc.cachefunc.cached_function(self, *args, **kwds)

Create a cached version of a function, which only recomputes values it hasn’t already computed. Synonyme: cached_function

INPUT:

  • f – a function
  • name – (optional string) name that the cached version of f should be provided with
  • key – (optional callable) takes the input and returns a key for the cache, typically one would use this to normalize input

If f is a function, do either g = CachedFunction(f) or g = cached_function(f) to make a cached version of f, or put @cached_function right before the definition of f (i.e., use Python decorators):

@cached_function
def f(...):
    ....

The inputs to the function must be hashable or they must define sage.structure.sage_object.SageObject._cache_key().

EXAMPLES:

sage: @cached_function
....: def mul(x, y=2):
....:     return x*y
sage: mul(3)
6

We demonstrate that the result is cached, and that, moreover, the cache takes into account the various ways of providing default arguments:

sage: mul(3) is mul(3,2)
True
sage: mul(3,y=2) is mul(3,2)
True

The user can clear the cache:

sage: a = mul(4)
sage: mul.clear_cache()
sage: a is mul(4)
False

It is also possible to explicitly override the cache with a different value:

sage: mul.set_cache('foo',5)
sage: mul(5,2)
'foo'

The parameter key can be used to ignore parameters for caching. In this example we ignore the parameter algorithm:

sage: @cached_function(key=lambda x,y,algorithm: (x,y))
....: def mul(x, y, algorithm="default"):
....:     return x*y
sage: mul(1,1,algorithm="default") is mul(1,1,algorithm="algorithm") is mul(1,1) is mul(1,1,'default')
True
sage.misc.cachefunc.cached_in_parent_method(self, inst, *args, **kwds)

A decorator that creates a cached version of an instance method of a class.

In contrast to CachedMethod, the cache dictionary is an attribute of the parent of the instance to which the method belongs.

ASSUMPTION:

This way of caching works only if

NOTE:

For proper behavior, the method must be a pure function (no side effects). If this decorator is used on a method, it will have identical output on equal elements. This is since the element is part of the hash key. Arguments to the method must be hashable or define sage.structure.sage_object.SageObject._cache_key(). The instance it is assigned to must be hashable.

Examples can be found at cachefunc.

sage.misc.cachefunc.cached_method(f, name=None, key=None)

A decorator for cached methods.

EXAMPLES:

In the following examples, one can see how a cached method works in application. Below, we demonstrate what is done behind the scenes:

sage: class C:
....:     @cached_method
....:     def __hash__(self):
....:         print "compute hash"
....:         return int(5)
....:     @cached_method
....:     def f(self, x):
....:         print "computing cached method"
....:         return x*2
sage: c = C()
sage: type(C.__hash__)
<type 'sage.misc.cachefunc.CachedMethodCallerNoArgs'>
sage: hash(c)
compute hash
5

When calling a cached method for the second time with the same arguments, the value is gotten from the cache, so that a new computation is not needed:

sage: hash(c)
5
sage: c.f(4)
computing cached method
8
sage: c.f(4) is c.f(4)
True

Different instances have distinct caches:

sage: d = C()
sage: d.f(4) is c.f(4)
computing cached method
False
sage: d.f.clear_cache()
sage: c.f(4)
8
sage: d.f(4)
computing cached method
8

Using cached methods for the hash and other special methods was implemented in trac ticket #12601, by means of CachedSpecialMethod. We show that it is used behind the scenes:

sage: cached_method(c.__hash__)
<sage.misc.cachefunc.CachedSpecialMethod object at ...>
sage: cached_method(c.f)
<sage.misc.cachefunc.CachedMethod object at ...>
class sage.misc.cachefunc.disk_cached_function(dir, memory_cache=False, key=None)

Decorator for DiskCachedFunction.

EXAMPLES:

sage: dir = tmp_dir()
sage: @disk_cached_function(dir)
....: def foo(x): return next_prime(2^x)%x
sage: x = foo(200);x
11
sage: @disk_cached_function(dir)
....: def foo(x): return 1/x
sage: foo(200)
11
sage: foo.clear_cache()
sage: foo(200)
1/200
sage.misc.cachefunc.weak_cached_function(self, *args, **kwds)

A version of CachedFunction using weak references on the values.

If f is a function, do either g = weak_cached_function(f) to make a cached version of f, or put @weak_cached_function right before the definition of f (i.e., use Python decorators):

@weak_cached_function
def f(...):
    ...

EXAMPLES:

sage: from sage.misc.cachefunc import weak_cached_function
sage: class A: pass
sage: @weak_cached_function
....: def f():
....:     print "doing a computation"
....:     return A()
sage: a = f()
doing a computation

The result is cached:

sage: b = f()
sage: a is b
True

However, if there are no strong references left, the result may be garbage collected, and thus a new computation would take place:

sage: del a
sage: del b
sage: import gc
sage: n = gc.collect()
sage: a = f()
doing a computation

The parameter key can be used to ignore parameters for caching. In this example we ignore the parameter algorithm:

sage: @weak_cached_function(key=lambda x,algorithm: x)
....: def mod_ring(x, algorithm="default"):
....:     return IntegerModRing(x)
sage: mod_ring(1,algorithm="default") is mod_ring(1,algorithm="algorithm") is mod_ring(1) is mod_ring(1,'default')
True

Previous topic

Bindable classes

Next topic

Fast and safe weak value dictionary

This Page