Bdtjtk

I want to use my own tokenizer to tokenize text stored as Dataset<String>, and get Dataset<Seq<String>> (so I can pass it to CountVectorizer).

Expected input (/tmp/fulltext.txt):

t1 t2 t3

t4 t5

Expected output:

[t1, t2, t3]

[t4, t5]

The tokenizer I write is (basically the thing it does now is the same as Tokenizer shipped with Spark, but I'll need to rewrite it to support tokenization of Chinese text, so I cannot use the official Tokenizer):

public class Utils {



  public static Seq<String> segment(String text) {

    String array = text.split(" ");

    List<String> tokens = new ArrayList<>();

    for (String term : array) {

      tokens.add(term.toLowerCase());

    }

    return JavaConverters

        .asScalaIteratorConverter(tokens.iterator())

        .asScala()

        .toSeq();

  }



}

The Spark application I'm trying to make is

public class TokenizeTest {



  public static void main(String args) {



    SparkSession spark = SparkSession

        .builder()

        .appName("Tokenize Test")

        .getOrCreate();



    Dataset<String> rawText = spark

        .read()

        .textFile("/tmp/fulltext.txt")

        .cache();



    Encoder<Seq> listEncoder = Encoders.bean(Seq.class);



    // Compilation error

    Dataset<Seq<String>> newText = rawText

        .map((MapFunction<String, Seq<String>>) s -> Utils.segment(s), listEncoder);



    newText.show();

    spark.stop();

  }

}

I'm a beginner of Spark, the above code is just what I think will work (after reading the official guide). But it turns out the code of TokenizeTest doesn't compile at all. Do you think there is a way to fix it?

Using Scala collections like this won't work. For once Seq is not Bean compatible, for second it is generic.

If you want split just use arrays with segement defined as:

public class Utils {



  public static String segment(String text) {

    return text.split(" ");

  }



}

and TokenizeTest defined as:

public class TokenizeTest {



  public static void main(String args) {



    SparkSession spark = SparkSession

        .builder()

        .appName("Tokenize Test")

        .getOrCreate();



    Dataset<String> rawText = spark

        .read()

        .textFile("/path/to/file")

        .cache();



    Encoder<String > listEncoder = spark.implicits().newStringArrayEncoder();





    Dataset<String > newText = rawText

        .map((MapFunction<String, String >) s -> Utils.segment(s), listEncoder);



    newText.show();

    spark.stop();

  }

}

In practice though, you might consider either org.apache.spark.sql.functions.split or org.apache.spark.ml.feature.Tokenizer instead of reinventing the wheel.