TheImpactof

GenerativeAIon

Talent,Technology,andHumanCapital

>

TABLEOFCONTENTS

3

INTRODUCTION:THEIMPACTOFGENERATIVEAIONTALENT9TECHNOLOGY9ANDHUMANCAPITAL

DavidFrancis,VP,Research,TalentTechLabs

5

PEOPLE-LEDANDTECHNOLOGY-POWERED:WALMART9SGENERATIVEAIJOURNEY

BenPeterson,GlobalLeaderofPeopleProduct,Walmart

MackenziCrank,GroupDirectorofPeopleStrategy&Innovation,Walmart

10

ACONVERSATIONWITHJOHNDICKERSON

JohnDickerson,Co-Founder&ChiefScientist,Arthur.ai

15

GENERATIVEAIINHIRING:BUILDTRUST

BEFOREBUSINESSIMPACT

EmreKazim,Co-Founder&Co-CEO,HolisticAI

20

CO-PILOTSINACTION

Phenom

Eightfold

Sense

26

NAVIGATINGAGENAIFUTURE

DavidFrancis,VP,Research,TalentTechLabs

2

3

TheImpactofGenerativeAIonTalent9

Technology9andHumanCapital

ByDavidFrancis,VP,Research,TalentTechLabs

NobelPrizewinningeconomistPaulKrugmanpredictedin1995(infamousforbeingcompletelywrong)that“ThegrowthoftheInternetwillslowdrastically,asthe?awin‘Metcalfe’slaw’—whichstatesthatthenumberofpotential

connectionsinanetworkisproportionaltothesquareofthenumberofparticipants—becomesapparent:mostpeoplehavenothingtosaytoeachother!By2005orso,itwillbecomeclearthattheInternet’simpactontheeconomyhas

beennogreaterthanthefaxmachine’s.”Theinternetwentontobecomelargelyresponsiblefortransforminghowwelive,shop,interactandconsumeinformation,nottomentionitsroleindisruptingentireindustriesandreshapingthebusinessclimate.Krugman'ssentiment,whilefunnyinhindsight,showshowdifficultitcanbetoseeinadvancethefullmagnitudeoftechnologicalshifts.

Todayweareinthe?rstinningofanothergeneration-de?ningbreakthroughintechnologythatwillsimilarlyreshapebusinessandsocietyandhavean

impactaslarge(andlikelyasyetunpredicted)astheintroductionoftheinternet.

ThatbreakthroughisgenerativeAI.Itistheclosestwe’vecometocomputersbeingableto“think,”“reason,”“create,”takeinstructionsand“work”onahumanlevelacrossanydomain.

Butwait.We’vehad“arti?cialintelligence”fordecades.Whyisthisparticular?avorsomuchdifferent?

Arti?cialintelligencehasbeena?eldofresearchsincethe1950s,anditshistoryhasbeenmarkedbyperiodsof

successivehopesanddisappointmentsinthe1980s,1990s,andthe2010s.GenerativeAI,whichwaspioneeredbyateamofGoogleResearchersin2017and“productized”byOpenAI’slaunchofChatGPT(whichbecamethefastest

growingconsumerproductinhumanhistory),isuniqueinthatitisa.)universallyaccessiblebyconsumersanddevelopers,b.)reallygoodatcomplexcognitivetasksacrossdomains.

ThesuccessofearlygenerativeAIapplicationshasledtoa?oodofinvestmentincompaniesandunderlying

infrastructure.Microsofthaspouredmorethan$10billionintoOpenAI.GoogleandAmazonhavecommitteda

combined$6billioninAnthropic(whileGooglecontinuestodevelopitsownmodels).Facebookisinvestingupwardsof$10billionbuildingtheworld’slargestarrayofgraphicsprocessorsintheworld(thechipsthatpowerAIapplications)

whileNVIDIAhasballoonedtoa$1.7trilliondollarmarketcapdesigningandsellingsuchchipstoeveryone.AI“wars”

arebeingfoughtacrossmultiplefronts:underlyinglargelanguagemodels(whichsoftwareandapplicationdevelopersrelyon),opensourceversusproprietary,computingpowerandtechnicalinfrastructure(whicharethe“rails”onwhichAIruns),andthebusinessapplicationsthatusegenerativeAItodriveresults.

4

McKinseypredictsthathalfofworkwillbeautomatedbygenerativeAIinthenextthreetofourdecades,while

GoldmanSachsestimatesitwillincreaseglobalGDPby$7trillion.OpenAIfounderSamAltmanwroteanarticle

proposingUniversalBasicIncomeandaland/equitytaxtoaddressthedisruptionheforeseesinthejobmarketasaresultofhuman-capableAI.CEOs,CTOs,andCIOsacrosstheworldaredemandingtheirteams?gurethisoutand

buildusecases.

Asaresult,nearlyeveryindustryiswrestlingwithhowtobestleveragegenerativeAItodriveresults,andthetalentindustryisnoexception.InthisissueoftheTrendsReport,weunpacktheuseofgenerativeAIacrossHRandtalentacquisition.Itisalookatwhatisinthemarketorcomingtomarketfromthosewhoareactuallybuildingit.

WebeginwithacasestudyledbyBenPeterson,HeadofPeopleProduct,andMackenziCrank,GlobalStrategyandInnovationLead,fromWalmart,thelargestemployerintheUS(andoneofthelargestintheworld),abouthowtheWalmartteambuiltandlaunchedanHRco-pilotwithgenerativeAIin60days.Walmartisonajourneytoreimagineassociateandcandidateexperiences,andemployerslargeandsmallalikecanlearnfromWalmart’sjourneyand

methodologyfornotjustembracinginnovationbutexecutingonitsvisionandgettingorganizationalalignmentfromtheCHROdown.

Wethenhaveabroad-ranginginterviewwithJohnDickerson,ChiefScientistatArthur.aiandProfessorofComputerScienceattheUniversityofMaryland,onmanyofthemorenuancedandtechnicalaspectsofmakinggenerativeAIworkinanenterprisecontext,particularlyinregulatedindustries.

Withnewcapabilitiescomenewrisksandvulnerabilities.EmreKazim,CEOofHolisticAI,anAIauditandriskmitigation?rm,discusseshowtomanagethenewformsofriskthatgenerativeAIbringstothetableandhoworganizations

shouldthinkaboutmitigatingsuchrisks.

OneoftheimmediateimpactsofgenerativeAIare“co-pilots,”generativeAIwidgetsthatareembeddedinexisting

softwareapplicationstomakethemeasiertouseorunlocknewlanguage-basedcapabilities.Therearealreadydozensofco-pilotsembeddedintoHRapplications.Wehighlightthree?rmsthathavedeployedgenerativeAIintheir

applications-Phenom,Eightfold,andSense-withaneyetowardshowtheybuilttheseco-pilotsandtheusecasestheyaddress,aswellasimpactfromtheclientsthathaveadoptedthesolutions.

Weendasusualwithadiscussionofimpactsandkeytakeawaysforvariousstakeholdersacrossthetalentcontinuum.Forreadersinterestedinthistopic,weencourageyoutoreadTalentTechLabs’broaderresearchongenerativeAI.

Wearetrulyinexcitingtimes,andwehopethisissueinspiresandinformsyouonyourowngenerativeAIjourney.

DavidFrancis

VP,Research,TalentTechLabs

5

People-ledandTechnology-powered:Walmart9sGenerativeAIJourney

ByBenPeterson,GlobalLeaderofPeopleProduct

MackenziCrank,GroupDirectorofPeopleStrategy&Innovation,Walmart

TTLINTRO:

Walmart,oneofthelargestandmostrespectedemployersontheplanet,builtanddeployeda

generativeAI-poweredemployeeassistant(“MyAssistant”)designedtoimprovetheproductivityofits50,000US-basedemployeesinjustover60days.Inthisarticle,twoleadersoftheinitiative,Ben

PetersonandMackenziCrank,describeWalmart’sGenAIjourney,howthey’vestayedapeople-led

organizationwhileusingtransformativetechnology,andwhatGenAImeansforthePeople/HRfunction.

PeopleoftenaskhowwedescribeWalmarttoday.

Tous,it’ssimple:Walmartisapeople-led,tech-powered

omnichannelretailerhelpingpeoplesavemoneyandlivebetter.

Wedothatbybeingthereforcustomershowevertheywanttoshop-instores,online,andthroughtheirmobiledevices.Eachweek,approximately240millioncustomersandmembersvisitapproximately10,500storesandnumerouseCommerce

websitesin19countries.

PoweringitallWalmart’sareapproximately2.1million

associatesworldwidewhobringWalmart’spurposetolife.

240MILLION

Customers

2.1MILLION

Associates

19

Countries

10,500

Stores

“…that’sthebestpart.We’reall

workingtogether.

That’sthesecret.

That’sthekey.”

Walmart’sfounder,SamWalton

WALMART9SGENERATIVEAIJOURNEY

“…that’sthebestpart.We’reallworkingtogether.That’sthesecret.

That’sthekey.”Walmart’sfounder,SamWalton,sharedthatquoteashewasreceivingthePresidentialMedalofFreedomin1992–just19days

beforehepassedaway.Itwastruethen,andit’struetodaythatpeopleareatthecenterofeverythingwedoatWalmart,andthat’showwe

orientedourapproachtogenerativeAI.

6

Walmarthasbeeninvestinginthiscoretechnologyforyears,understandingthatGenAIwouldbeapowerfultoolnotjustforthecustomerexperience,buttoelevatethedailyworkexperienceofWalmart’sassociatesaroundtheworld.

Nearlyallcompaniesinvestintechthat’sfocusedontheconsumerexperience-farfewerbringthatsameleveloffocustotheirownworkforce.AtWalmart,weviewthetwoasinextricablylinked.

Asourteamfocusedoncustomerandbusinessapplications,wealsohadaclearobjectivetoputthepowerofgenerativeAIinthehandsofeverycampusassociate.

THEPATHFROMVISIONTOVIABLE

Ourvisionwasambitious:toreimaginetheexperienceforWalmart’s2.1millionassociatesbybuildingintuitive,

consumer-gradeexperiencesthatunlockthefullestpotentialofourassociates.Thatmeansprovidingtechnologycapabilitiesthatcanenhanceandstreamlineemployees’digitalinteractionsacrosshiring,onboarding,learning,performancemanagement,careerdevelopment,compensation,bene?ts,payroll,andmore.

Tothatend,Walmartisonajourneytocombinenearly300disparatesystemsintoasingleexperiencelayerfor

associates.Oursuperappforcampusassociates(wealsohaveversionsforourfrontlineassociatestomeettheirspeci?cneeds)isknownasMe@Campusandcontainsawidevarietyoftoolsforassociates–includingthe

GenAI-powered“MyAssistant.”

Thenamesaysitall.MyAssistantisdesignedtohelpWalmartassociatessavetimethroughouttheirday,makingtheirworksmootherandallowingthemtofocusonthemosthuman-centrictasks,asopposedtomundaneworklike

summarizingmeetingnotesthey’djusttaken.

TheteambeganwithafocusonWalmart’sHomeOfficeassociatesintheU.S.Similarprojectsmighthavebegunwithasmallusergrouptotest,iterateandpilotuntilthetoolwasnearlycomplete.Itwouldhavebeenthesaferchoice,butitwouldhavemeantkeepingthistechnologybehindcloseddoors.Theteamtookaradicalapproach:putthetoolin

thehandsof50,000campus-basedassociatesandempowerthemtostartlearningandreapingtherewardsofGenAI.

ThankstoWalmart’sstrongtechfoundation,teamsfromproduct,engineering,datascience,design,andthebusinesscollaboratedtobringMyAssistantfromvisiontoviableinjust63days.

BUILDINGAPEOPLEPRODUCTORGANIZATIONFORTHEFUTURE

Creatingworld-classexperiencesforWalmart’sassociatestakesateamwithabroadrangeofexpertise,laser-focusedonasharedmission.Inearly2023,Walmart’sleadersdecidedtotakeanewapproachtoproductmanagementinthePeoplespace–onethatkeptthecriticalconnectionsinplacebutallowedforanewkindofstrategicalignment.

7

DonnaMorris,Walmart’sChiefPeopleOfficer,isalongtimeadvocateforthecriticalroleoftechnologyinde?ningthe

associateexperience.BringingthePeopleProductteamintoherorganizationmeantgivingthemadirectlineofsighttoDonna'svisionandstrategyfortheentirepeoplefunction,whichallowedtheProductteamtomovewithspeed,look

aroundcornersanddeliverproductsdesignedforassociatestodayandwellintothefuture.

Ofcourse,thosesameconnectionsthathelpedbringMyAssistanttoliferemaincritical–engineers,datascientists,designersandbusinessexpertsworkinclosepartnershipwiththePeopleProductteamtobringthevisiontolifeinserviceofWalmart’sassociatesandbusiness.

CHANGEMANAGEMENT

GenerativeAIisaparadigmshiftingtoolandleadingaworkforcethroughtheintegrationofGenAIintotheirdailyworktakestime.ThinkabouttheadventofMicrosoftExcelinthe1980s.Earlyuserswerewary–largelybecausetheydidn’t

fullyunderstandhowtoharnessthepoweritrepresented.Usersneededtolearnthesigni?canceofaPivotTableandVLOOKUPformulas,justastoday’sworkforceneedstounderstandpromptingandhowtocreatehigh-impactuse

cases.

AtWalmart,weuseacombinationofformalandinformallearning.Theapproachisrootedinbuy-infromsenior

leaders,whoareencouragedtounderstandthetechnologythemselvesandchampionitfortheirteams.Someofthemosteffectivechangemanagementpracticesaresimplyhostingorjoiningexistingtownhallstodemonstrate

pragmaticusecases.Generally,oncepeoplehaveseenthetoolinaction,they’llovercometheirownhesitationandmoveintoanexploratoryphase.

That’swhereformaltrainingcomesin.Webelieveindeliveringhands-ontrainingtomanagerstoensuretheyhavetheknowledgeandexperiencetodevelopusecaseswithintheirteams.BecauseGenAImovessoquicklyandlearnsbasedonwhatusersinput,wesoughtthebroadestpossibleapproachthatcouldbedeliveredsecurelyandwithanexcellentuserexperience.Wecontinuetobuildonthisapproach,includingrecentlylaunchingadedicatedlearningexperience

withinMyAssistantcalled“LearntoPrompt,”whereassociatescanreceivedetailedfeedbackandsuggestionstoimprovetheirinputsfromthetoolitself.

SimilartoExcel,therewillalwaysbeacombinationofsuper-users,pro?cientusers,beginnersandlateadopters.The

keyistounderstandtheneedsofeachgroup,continuedemonstratingreal-lifevalueandprovideaframeworkforthemtoengagewheretheyare.

WeencourageleaderstotalktotheirteamsaboutMyAssistantandhighlightthewaysitappliestothemspeci?cally.

Seeingisbelieving,andweknowhands-onexperiencewiththetechnologyisthemosteffectivewaytolearn.We’realsocreatingaculturewherethere’snostigmaaroundusingMyAssistantasastartingplace.Werecognizeand

appreciatethatifanassociateneedstowriteawhitepaper,forinstance,theycansavevaluabletimeontheir?rst

draftbyusingMyAssistant,leavingmoreenergyforcreativityandre?nement,resultinginastrongerproductoverall.

8

GENERATIVEAIFORPEOPLEFUNCTIONS

We’restillscratchingthesurfaceofwhatgenerativeAIcanunlockforourcustomers,members,associatesandourbusiness.OurPeopleteammembersaresomeofourstrongestusersofMyAssistant,andthey’reconstantly

discoveringnewusecasesspeci?ctotheirpartsofthebusiness.Someusecaseswe’veseen:

Creatinga?rstdraft

ofanexperientiallearning

activity.

Draftingindividual

developmentplanswithafocusonaspeci?crole.

Ideatingaroundinterview

questionsbasedonajob

description.

Andthosearejustafewexamples–everydayacrossthebusiness,thousandsofusersturntoMyAssistanttotrynewthings.Thecommonalityacrosstheseusecasesisthattheyalwayshaveahumanintheloop–ittakesthatspecial

levelofHRexpertisetounderstandwhatmakesatrulyimpactfullearningactivity.Ittakescontextaboutourbusinesstore?neadevelopmentplan.Andittakesempathyandstrategicthinkingtocraftanddeliverinterviewquestionsthattrulyunlockthebesttalentforourorganization.

Thatwillcontinuetobetrue,bothasthetechnologyevolves,andasourownusersbecomemoresophisticated.We’llbeabletodriverealefficiency,notforthesakeofefficiencyalone,butwiththegoalofhelpingourassociatesleanintotheareaswherethey’reuniquelypositionedtomakeanimpact.ThatwillbetrueinHRandbeyond.

LOOKINGAHEAD

AsGenAIcontinuestodevelop,onethingisclear:companieshaveachoicetomake.Usetechnologytoruthlesslycutcosts,orintentionallydeploytechnologyinawaythatbringsyourpeopleonthejourney.AtWalmart,we’vechosenthesecondpath.We’reapeople-led,tech-poweredomnichannelretailer,andweknowthatnomatterhowtechnology

evolves,itwillalwaysbeourpeoplewhomakethedifference.

9

ABOUTTHEAUTHORS

BenPeterson

GlobalLeaderofPeopleProduct,Walmart

BenPetersonisthegloballeaderofWalmart’sPeopleProductorganization,oneoftheteamsbringing

Walmart’speople-led,tech-poweredvisiontolife.Heisresponsiblefortheproductvision,strategyanddeliveryofworld-classdigitalexperiencesforWalmart’s2.1millionassociatesacrossWalmartUS,Sam’sClub,WalmartInternationalandHomeOffice.UnderBen’sleadership,PeopleProducthasledthe

strategyandlaunchofinnovativenewproductssuchasMe@Campus,MyAssistantandMe@SamsClub.PriortojoiningWalmart,BenspentadecadeatDeloitteConsulting,whereheco-foundedDeloitte’s

productmanagementconsultingpracticeandbuiltmarket-leadingdigitalproductswithFortune500companies.

MackenziCrank

GroupDirectorofPeopleStrategy&Innovation,Walmart.

MackenziCrankisGroupDirectorofPeopleStrategy&InnovationatWalmart.SheleadsanewfunctionfocusedondrivingchangeinWalmart’sPeoplefunctionanddeliveringinnovativesolutionstolistentoWalmart’sworkforceandimprovetheassociateexperience.Mackenzihasspentmostofhercareerinappliedresearchandconsultingrolesfocusedondeliveringorganizationalandcustomerinsights.

ABOUTWALMART

WalmartInc.(NYSE:WMT)isapeople-led,tech-poweredomnichannelretailerhelpingpeoplesavemoneyandlivebetter–anytimeandanywhere–instores,online,andthroughtheirmobiledevices.Eachweek,

approximately240millioncustomersandmembersvisitapproximately10,500storesandnumerouseCommercewebsitesin19countries.With?scalyear2023revenueof$611billion,Walmartemploysapproximately2.1

millionassociatesworldwide.Walmartcontinuestobealeaderinsustainability,corporatephilanthropy,andemploymentopportunity.AdditionalinformationaboutWalmartcanbefoundbyvisiting

,onFacebookat

/walmart

,onX(formerlyknownas

Twitter)at

/walmart

,andonLinkedInat

/company/walmart/

.

walmart、、?

*Note:ThisreportwassubmittedinJanuary2024.

10

AConversationwithJohnDickerson

JohnDickerson,Co-founder&ChiefScientist,Arthur.ai

INTRO:

JohnDickersonisatenuredprofessorofComputerScienceattheUniversityofMarylandandco-founderandChiefScientistofArthur.ai.Hisresearchliesattheintersectionofmachinelearningandeconomics,withafocusondesigningincentivesfor“good”participationinhighlycomplexsystems.Inthis

conversation,wediscussmanyofthenuancesandmoretechnicalaspectsofworkingwithanddeployingalargelanguagemodelinanenterpriseenvironment.

*Note:Theinterviewwasconductedliveandeditedforbrevity.

DavidFrancis

VPResearch,TalentTechLabs

Forthosewhohaven'tmetyou,canyouintroduceyourselfandArthur.ai?

JohnDickerson

Co-Founder&ChiefScientist,Arthur.ai

Arthur.aiwasfoundedabout4.5yearsago.WestartedinWashingtonD.C.butwe'reinNewYorknow.OurCEOhadstartedacompanythatwasacquiredintoCapitalOnewherehebuiltouttheirinternalAIteamtoafewhundredfolks,buildingthingslikecreditdecisioningmodels,etc.Theideaforthecompanywasthatwe

shouldhavehuman-centeredbutautomatedmonitoringofAImodelsthatareoutinproductionand

supportingdecisionmaking.We'remodelandindustryagnostic;alotofourclientsarein?nance,health,manufacturing,andafewinHR.

Recently,we'vemovedintomonitoringLLMs(largelanguagemodels),whichtakesafewforms.Wehavea

productthatprotectsagainstmalicioususerslikethefolksthataretryingtodothingslikemaketravelplansonyourpublicfacingcar-buyingLLM,ortryingtogetyourLLMtodosomethingit’snotsupposedtoor

otherwiseharmingthebrand.WealsohaveaproductthathelpsevaluateandselecttheappropriateLLM

fromthelargenumberofmodelsthatareouttherebasedondifferentusecasesandneeds.MybackgroundisI'matenuredcomputerscienceprofessorattheUniversityofMaryland,andIwasfocusedonfairnessandrobustnessofdynamicsystems,andmetmyco-founderashewasbuildingouttheproduct.

DavidFrancis

VPResearch,TalentTechLabs

HowareLLMsandgenerativeAIdifferentfrom"traditional"AImodelsofyesterday?

11

JohnDickerson

Co-Founder&ChiefScientist,Arthur.ai

Generativetasksaren'tnew,forexampleit'sfairlycommonincomputervision.Whatisnewisacouple

things.First,thesemodelsworkreallywellnow.Partofthatisyoucandoabunchofworkupfrontto

"pre-train"amodelonalotofdata(the"p"inGPT),soyoufeedamodelahugesetofdataandonceits

predictions(sayforguessingwhichwordshouldcomenextinasentence)areprettygoodyoucanpublishittotheworld.Anotherotherbigdifferenceisthesizeofmodels.Thesemodelshavebeenpre-trainedonanenormousamountofdata,thoughitisreallyexpensivebothonthetrainingside(whatitcoststotrainthe

modelinitially)andinferencing(whatitcoststousethemodeltoe.g.respondtoprompts).Forawhiletherewasthisideaofjustaddingmoreandmoredataforbetterandbetteroutcomes,butwe'vegottentothe

pointwherecompaniesarethinkingaboutmaybesmallermodelsbettersuitedforspeci?cusecases.Last,

thebiggestdifferencefrombeforeisthatweIvereallydemocratizedaccesstotheseenormous,powerfulmodelswhichisnew.TheOpenAIAPIforexampleissoeasytouseandthere'sneverbeenthislevelof

access.

DavidFrancis

VPResearch,TalentTechLabs

Thestrategy,atleastamongtheGoogle'sandOpenAI'softheworld,seemstobelaunchingnewfoundationalmodelswithanincreasinglylargerandlargertrainingdataset.Butthereseemstobediminishingreturnsintermsofperformance.Whatdoesthefutureholdintermsofbreakthroughsinrawfoundationalmodel

capabilities?

JohnDickerson

Co-Founder&ChiefScientist,Arthur.ai

Thisisaninterestingquestionespeciallyfor2024,andbasicallyisabouthowwellthesemodelsscale.For

thoseinterestedinkindofamoretechnicaldiscussion,thebestplacetostartistolookupChinchilla'sScalingLaws.Ithinkthejuryisstilloutonthemarginalreturnofincreasingthesizeofthesemodels.Forexample

ChatGPT-3was175billionparameters,andGPT-4isapparentlyaroundatrillion.Thatsaid,OpenAIisnot

transparentaboutthestructureoftheirmodel,andalotoffolksbelievethatinfactwhatthey'vedoneismeshtogetherabunchofsmaller"expert"modelsintoonecohesivewhole,andmaybeit’sstillatrillionparameters

total,butit’snotone"general"modelbutabunchofsmalleronesworkingtogether.

Thechallengeisthere'sahugecosttotrainingandrunningtheseenormousmodels.Whatwe'veseenistherearealotofcaseswhereyoucanactuallygetbetterperformance(atcheapercost)withasmallermodel

designedforaspeci?cusecase,andalotoftimesusingageneralistmodellikeChatGPTforeverythingisactuallyreallyexpensive.

AgoodexampleisamodelrecentlyreleasedbyMicrosoftcalledPhi-2,whichisextremelygoodatcodingtasks,isverysmall,butisnItasgreatatothertasks.Soforitsspeci?cusecase(coding)itIsordersof

magnitudecheaperthanageneralistmodel,butyouwouldn'twanttobeusingitsayforwritingcreativeemails.

12

DavidFrancis

VPResearch,TalentTechLabs

Let'sworkthroughsomeofthedecisionmakingthatneedstohappenwhenchoosingLLMinfrastructure.

Whataresomeguidingprincipleswhendecidingonaparticularapproachand/ormodel?

JohnDickerson

Co-Founder&ChiefScientist,Arthur.ai

Thereareabsolutelydifferencesbetweenthesemodels.IIllsayupfront,andthisiscomingfromsomeonewhoisbuildingproductstoevaluatethesemodels,evaluationforLLMsisextremelyhard.There'sreallynogoodholisticLLMevaluationtooloutthererightnow.Alotofevaluationsuites,includingourstosome

degree,focussolelyonjusttheinputsandtheoutputs.Whereaswhatyou'dreallylikeistofocusonthewholesystem.

ThewayIthinkaboutmodelselectionisyou?rstneedtolookatyourusecases,andthenseewhichmodelormodelsaregoingtoworkforthatusecase,andthencompareitagainsttheobviously?niteresourcesthateveryorganizationcontendswith.Soforexample,itcouldbethatGPT-4Turboknocksalltheothersoutof

thewateringeneral,exceptformyonespeci?ctask,sayit’ssortingresumes,orextractingdata,andso

maybeit'sbettertouseasmallermodelthatwillbealotcheaperforthattask,andbythewayifit'sopen

sourceIcanrunitlocallyandseeexactlyhowitworksversusforexamplethatmuchlargerbutmoreopaquemodelthatismoreexpensive(andwedon'treallyknowifandwhenthecompanybehinditwillchangethe

modelweightsatsometimeinthefuturewhichwillimpacthowitperforms).

Thebestorganizationsuseacombinationofmodelsbestsuitedfortheirparticulartasks,butthatworktogetherforaseamlessexperience.

DavidFrancis

VPResearch,TalentTechLabs

WhataretheinformationsecurityimplicationsandrisksassociatedwithLLMs?Manyareconcernedaboutpromptsbeingusedtotrainfuturemodelsandthepossibilityfore.g.IPleakage.

JohnDickerson

Co-Founder&ChiefScientist,Arthur.ai

Therecertainlyaresomerisks.Ifyouuseanopensourcemodelorrunyourownmodellocally,withyourowndata,onyourowninfrastructure,thenobviouslythere'snorisk.SoforexampleyougotoHuggingfaceand

downloadamodelandhostitonyourinfrastructureusingyourowndata.IfyouusesayChatGPTviathewebbrowser,thenthosepromptscanbeusedtotrainfutureiterations,andthepromptinfoandresponsecan

becomepublic.Totheircredit,youcanaccessChatGPTviaMicrosoftinaVPC(virtualprivatecloud),whichisa?rewalledinstanceandthereIsnodataegress,whichishowalotoflargeenterprisesobviouslyaregoingtoaccessthesecapabilities.Allthebigplayersatthispointcandothat,butOpenAIhastheadvantageof

13

workingcloselywithMicrosoft,andMicrosoftisthekingofworkingwithenterprises.

Anotherpotentialriskistheinabilitytoaccesstheunderlyingtrainingdatasetforcertainregulatedactivities.Weworkwithalotofbanksand?ntechcompaniesthatareusingAItodothingslikemaketradingdecisions.Nowit’spossibleatsomepointtheymightneedtoexplainwhyadecisionwasmade,ortheymaygetsued,

andtheyhavetopointatsomething.Andobviouslyyoucan'tseethetotalityofe.g.OpenAI'strainingdata.Sorecentlywe'restartingtoseemodelsreleasedthatincludeboththemodelandthe