.NET for Apache Spark hits v1.0
source link: https://cloudblogs.microsoft.com/opensource/2020/10/30/announcing-net-apache-spark-1/?ocid=AID2423380_TWITTER_oo_spl100001599547071
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
.NET for Apache Spark hits v1.0
About two years ago, we heard an increasing demand from the .NET community for an easier way to build big data applications with .NET, outside of needing to learn Scala or Python. Thus, in a collaboration between Azure Data and .NET teams, we started the .NET for Apache® Spark™ open source project.
Today, we are happy to announce the release of version 1.0 of .NET for Apache Spark, an open source package that brings high performance APIs for using Apache Spark from C# and F#. Indeed, it is the same technology that powers the .NET experience for Apache Spark inside Azure Synapse Analytics.
At a high-level, .NET for Apache Spark offers users the ability to:
- Write Spark applications in C# or F# targeting .NET Core 2.1+.
- Reuse existing C#/F# libraries and use the power of Apache Spark to parallelize them.
- Leverage the high-quality Visual Studio or Visual Studio Code IDEs for building Spark apps.
The new .NET for Apache Spark v1.0 brings in additional capabilities to an already rich library:
- Support for DataFrame APIs from Spark 2.4 and 3.0.
- 47 new Spark SQL functions to support Spark 3.0 and 464 Spark SQL functions for Spark 2.4, bringing the total to 511 out-of-box Spark SQL functions to make it even easier for you to author Spark apps.
- First-class support for the Linux Foundation’s Delta Lake, Microsoft’s Hyperspace Indexing Sub-system, ML.NET, and support for Apache Spark’s MLLib functionality.
- Community-contributed Docker images that allow you to try and debug .NET for Apache Spark in a single-click, play with it using .NET Interactive notebooks, as well have a full-blown local development environment in your browser using VS Code so you can contribute to the open source project, if that’s of interest to you.
We are extremely grateful to everyone who contributed towards reaching this important milestone. For more about the history of the project, key contributors, and its use in production pipelines, read the full announcement.
To learn more about .NET for Apache Spark, check out our presentation at the Databricks’ Spark+AI Summit 2019, Microsoft Build 2019, SQLBits 2020, and the demo at Ignite 2020.
Learn more about .NET for Apache Spark:
- Check out the .NET for Apache Spark code on GitHub.
- Ready to try this out? Check out getting started.
- Feel like contributing? Start with the outstanding issues.
Don’t miss our presentation describing this major release in the upcoming .NET Conf 2020: “The Missing Piece: Diving into the World of Big Data with .NET for Apache Spark”!
Questions or feedback? Let us know in the comments below.
Recommend
-
63
Realtime predictions with Apache Spark/Pyspark and Python There are many blogs that talk about Apache spark and how scalable it is to build Machine Learning models using Big data. But, there are few blogs that...
-
66
Editor’s Note: Alastair Green (Neo4j Query Languages Lead) and Martin Junghanns (Neo4j Cypher for Apache Spark project) will be
-
27
What is Apache Spark? Apache Spark is an in-memory distributed data processing engine that is used for processing and analytics of large data-sets. Spark presents a simpl...
-
40
Spark , The word itself is enough to generate a spark in every Hadoop engineer’s mind. A n in-memory processing tool
-
68
本文资料来自2019-03-24在上海举办的 Hadoop+Spark生态技术开放日。参见：https://cloud.tencent.com/developer/salon/salon-1114。 分享者：邵赛赛，腾讯云专家工程师。Apache Spark 以及 Apache Chukwa 的 Committer，Apache L...
-
53
Why We Need Apache Spark With immense amounts of data, we need a tool to rapidly digest it Eric Girouard ...
-
60
High Level Overview of Apache Spark What is Spark? Let’s take a look under the hood Eric Girouard
-
25
April 24th, 2019 Today at
-
14
Announcing Version 1.0 of .NET for Apache SparkAnnouncing Version 1.0 of .NET for Apache Spark
-
4
Description .NET for Apache Spark empowers .NET developers to participate in the world of big data analytics. In this episode, Jeremy chats with Michael Rys to discuss some of the new features and capabilities availa...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK