[SciPy-User] scipy sparse limits

[Eric]

Hi, I'm trying to use large 10^5x10^5 sparse
matrices but seem to
be running up against a scipy limit:

n=10**5, x=sp.rand(n,n,.001) gets
"ValueError: Trying to generate a random sparse matrix such
as the product of dimensions is greater than
2147483647 - this is not supported on this machine"

Does anyone know why limit is there and if I can avoid it?
(fyi, I'm using a macbook air with
4gb memory and the enthought distribution)

thanks,
Eric

[Pauli Virtanen]

12.03.2012 17:48, Eric kirjoitti:
[clip]

Post by Eric
n=10**5, x=sp.rand(n,n,.001) gets
"ValueError: Trying to generate a random sparse matrix such
as the product of dimensions is greater than
2147483647 - this is not supported on this machine"
Does anyone know why limit is there and if I can avoid it?

The limit seems to be only in the rand() routine --- you can create
larger random matrices otherwise.

The limit probably shouldn't be there --- this is probably a bug. As a
workaround, you can copy and paste the rand() routine to your own code
and remove the size check:

https://github.com/scipy/scipy/blob/master/scipy/sparse/construct.py#L573

[Chris Weisiger]

Hi,  I'm trying to use large 10^5x10^5 sparse
matrices but seem to
n=10**5, x=sp.rand(n,n,.001) gets
"ValueError: Trying to generate a random sparse matrix such
as the product of dimensions is greater than
2147483647 - this is not supported on this machine"
Does anyone know why limit is there and if I can avoid it?

log2(2147483648) = 31, so it sounds like you're running into a 32-bit
limitation somewhere. If you can, using 64-bit Python/Numpy/Scipy
would probably get you around this.

-Chris