Bdtjtk

This might happen inside scikit, and it depends on what you're doing. I recommend reading the documentation for the functions you're using. You might be using one which depends e.g. on your matrix being positive definite and not fulfilling that criteria.

EDIT: How could I miss that:

np.isnan(mat.any()) #and gets False

np.isfinite(mat.all()) #and gets True

is obviously wrong. Right would be:

np.any(np.isnan(mat))

and

np.all(np.isfinite(mat))

You want to check wheter any of the element is NaN, and not whether the return value of the any function is a number...

I got the same error message when using sklearn with pandas. My solution is to reset the index of my dataframe df before running any sklearn code:

df = df.reset_index()

I encountered this issue many times when I removed some entries in my df, such as

df = df[df.label=='desired_one']

This is the check on which it fails:

• https://github.com/scikit-learn/scikit-learn/blob/0.17.X/sklearn/utils/validation.py#L51

Which says

def _assert_all_finite(X):

    """Like assert_all_finite, but only for ndarray."""

    X = np.asanyarray(X)

    # First try an O(n) time, O(1) space solution for the common case that

    # everything is finite; fall back to O(n) space np.isfinite to prevent

    # false positives from overflow in sum method.

    if (X.dtype.char in np.typecodes['AllFloat'] and not np.isfinite(X.sum())

            and not np.isfinite(X).all()):

        raise ValueError("Input contains NaN, infinity"

                         " or a value too large for %r." % X.dtype)

So make sure that you have non NaN values in your input. And all those values are actually float values. None of the values should be Inf either.

This is my function (based on this) to clean the dataset of nan, Inf, and missing cells (for skewed datasets):

import pandas as pd



def clean_dataset(df):

    assert isinstance(df, pd.DataFrame), "df needs to be a pd.DataFrame"

    df.dropna(inplace=True)

    indices_to_keep = ~df.isin([np.nan, np.inf, -np.inf]).any(1)

    return df[indices_to_keep].astype(np.float64)

I had the error after trying to select a subset of rows:

df = df.reindex(index=my_index)

Turns out that my_index contained values that were not contained in df.index, so the reindex function inserted some new rows and filled them with nan.

I had the same error, and in my case X and y were dataframes so I had to convert them to matrices first:

X = X.as_matrix().astype(np.float)

y = y.as_matrix().astype(np.float)

With this version of python 3:

/opt/anaconda3/bin/python --version

Python 3.6.0 :: Anaconda 4.3.0 (64-bit)

Looking at the details of the error, I found the lines of codes causing the failure:

/opt/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py in _assert_all_finite(X)

     56             and not np.isfinite(X).all()):

     57         raise ValueError("Input contains NaN, infinity"

---> 58                          " or a value too large for %r." % X.dtype)

     59 

     60 



ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

From this, I was able to extract the correct way to test what was going on with my data using the same test which fails given by the error message: np.isfinite(X)

Then with a quick and dirty loop, I was able to find that my data indeed contains nans:

print(p[:,0].shape)

index = 0

for i in p[:,0]:

    if not np.isfinite(i):

        print(index, i)

    index +=1



(367340,)

4454 nan

6940 nan

10868 nan

12753 nan

14855 nan

15678 nan

24954 nan

30251 nan

31108 nan

51455 nan

59055 nan

...

Now all I have to do is remove the values at these indexes.

i got the same error. it worked with df.fillna(-99999, inplace=True) before doing any replacement, substitution etc

In my case the problem was that many scikit functions return numpy arrays, which are devoid of pandas index. So there was an index mismatch when I used those numpy arrays to build new DataFrames and then I tried to mix them with the original data.

If you can't find the problem in X, check in y