Neuralink Interview #5 - Breck Yunits
source link: https://docs.google.com/presentation/d/1CgTbCPSYdKY2_oSvqV63GueIlUggnfdwGJhYgEvMTZI/edit#slide=id.g79162dd456_0_98
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Neuralink Interview #51Building“CoreTable”andtheOurWorldinData“DataExplorer”BreckYunits-June2,20222Timeline&ContributionsMax,Hannah,Esteban,Joe,Edouard,Charlie,Diana,Bobbie—theresearchteam–responsibleforeditorial,content,anddata.MatthieuresponsibleforWordpress.Jaiden&DanielresponsibleforGraphercode.Max&ErnstdesignofDataExplorerUI.HannahcreditforDataExplorerDSL.ShahidcreditforDataExplorerbuilderUX.IwasresponsibleforDataExplorercode,designandimplementationofCoreTable,andrefactorofGrapher.WithcriticalsupportfromDaniel&Marcel.3StateofOurWorldinData,March2020Pros-Researchteam;reputation;belovedchartsCons-Annualdataonly;40klocw/0tests;sluggishdevspeed4DucktapingGraphertoshowdailycharts56Covid-19DataExplorer73fundamentalproblems1.Nodataarchitecturefromrootthinking2.Notestsandnomodularity3.Researchersneededengineerstocreateandshipexplorers8Fixingthedataarchitecture:CoreTablePreviouslyourdatatransformswereamixofdatastructures,adhoc,duplicated,untestedandintertwinedwithchartlayoutcode.CoreTablewasournew“PandasforTypeScript”.StandalonelibrarythatdecoupledallchartlogicfromdatalogicImmutabledataflowlibrary.Eachoperationcreatesanewtable(lazily).Tablesallthewaydown.9OurdataRelativelysmalltabulardata<1PB.1,000’sofdatasourcesofvaryingquality.10,000’sofdatasets.100,000’sofcolumns.~1Brows.Lotsofhumaneditedmetadata(provenance,attribution,explanation,etc).StoredinmixofMySQLandCSVs.10CoreTable:Overview
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK