Bdtjtk

I am using this piece of code to calculate spark recommendations:

    SparkSession spark = SparkSession

            .builder()

            .appName("SomeAppName")

            .config("spark.master", "local[" + args[2] + "]")

            .config("spark.local.dir",args[4])

            .getOrCreate();

    JavaRDD<Rating> ratingsRDD = spark

            .read().textFile(args[0]).javaRDD()

            .map(Rating::parseRating);

    Dataset<Row> ratings = spark.createDataFrame(ratingsRDD, Rating.class);

    ALS als = new ALS()

            .setMaxIter(Integer.parseInt(args[3]))

            .setRegParam(0.01)

            .setUserCol("userId")

            .setItemCol("movieId")

            .setRatingCol("rating").setImplicitPrefs(true);



    ALSModel model = als.fit(ratings);

    model.setColdStartStrategy("drop");

    Dataset<Row> rowDataset = model.recommendForAllUsers(50);

These are maven dependencies to make this piece of code work:

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>2.4.0</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-sql_2.11</artifactId>

        <version>2.4.0</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-mllib_2.11</artifactId>

        <version>2.4.0</version>

    </dependency>

    <dependency>

        <groupId>org.scala-lang</groupId>

        <artifactId>scala-library</artifactId>

        <version>2.11.8</version>

    </dependency>

Calculating recommendations with this code takes ~70sec for my data file. This code produces following warning:

WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS

WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS

WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK

WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK

Now I try to enable netlib-java by adding this dependency in maven:

    <dependency>

        <groupId>com.github.fommil.netlib</groupId>

        <artifactId>all</artifactId>

        <version>1.1.2</version>

        <type>pom</type>

    </dependency>

to avoid crashing of this new environment I had to do this extra trick:

LD_PRELOAD=/usr/lib64/libopenblas.so

Now it also works, it gives no warnings, but it works slower and it takes ~170sec on average to perform the same calculation. I am running this on CentOS.

Shouldn't it be faster with native libraries? Is it possible to make it faster?