This dissertation is divided into two parts, the first of which revolves around rates of convergence to equilibrium for single-server queues (consisting of Chapters 2 and 3) and the second of which revolves around asymptotic variability and large deviations for departure processes in single-server queues (consisting of Chapters 4 and 5). Throughout the thesis, significant emphasis is placed on the Brownian modeling of queues, using reflected Brownian motion (RBM) both as a direct modeling tool as well as vehicle through which to obtain insights into pre-limit queueing behavior.

Prior research on how firms in developing countries undergo internationalization has been limited. However, as more firms have successfully internationalized in recent years, there is a need for better understanding why, how, and where they choose to internationalize. This dissertation examines the internationalization of business groups in developing countries and the importance of institutional settings. Based on this examination, this dissertation advances the thesis that business groups have two paths that need to be balanced in order for business groups to thrive. The first path is to enter other institutionally close developing countries by exploiting the firm's existing resources and know-how. The second path is to enter institutionally close developed countries to acquire technological sophistication and improve their organizational learning capabilities through long-term investments. This dual approach enables business groups to survive economic downturns and outperform established industry leaders as seen during the recent financial crisis. In support of this thesis, I incorporate several prior theories on the liability of foreignness and institutional theory; investigate the role of culture in the internationalization of business groups; and analyze two in-depth case studies. This multi-pronged approach provides a framework for scholars to better understand the internationalization and associated strategic motives. This study contributes to the theory of internationalization and to the ongoing research of culture in the field of international business. Lastly, this study makes valuable recommendations to managers and government policymakers on how to optimize the resources and capabilities of business groups to support their domestic markets.

We introduce an online learning platform that scales collaborative learning. We study the impact of team formation and the team formation process in massive open online classes. We observe that learners prefers team members with similar location, age range and education level. We also observe that team members in more successful teams have diverse skill sets. We model the team formation process as a cooperative game and prove the existence of stable team allocations. We propose a polynomial-time algorithm that finds a stable team allocation for a certain class of utility functions. We use this algorithm to recommend teams to learners. We show that team recommendations increase the percentage of learners who finish the class.

In this dissertation, I investigate executives' attention to competitors and the implications for firm innovation. Using a hand-collected dataset on competition, innovation, and executive experience in the full population of public U.S. enterprise infrastructure software firms, I examine attention to competitions in three empirical papers. In the first paper, I show that attending to competitors that activate opportunity-based rather than threat-based views of competition has a positive relationship with product innovation. In the second paper, I show that executives are more attentive to competitors at the periphery of an industry when the experience of the executive team amplifies (i.e. mirrors) the experience of the CEO. In the third paper, I use social network analysis to show that personal similarities between CEOs can influence competition between firms. As a whole, my dissertation suggests that executives can be strategic in how they think about competition, with tangible benefits for firm performance.

As electric sector stakeholders make the decision to upgrade traditional power grid architectures by incorporating smart grid technologies and new intelligent components, the benefits of added connectivity must be weighed against the risk of increased exposure to cyberattacks. Therefore, decision makers must ask: how smart is smart enough? This dissertation presents a probabilistic risk analysis (PRA) framework to this problem, involving systems analysis, stochastic modeling, economic analysis, and decision analysis to quantify the overall benefit and risk facing the network and ultimately help decision makers formally assess tradeoffs and set priorities given limited resources. Central to this approach is a new Bayes-adaptive network security model based on a reformulation of the classic "multi-armed bandits" problem, where instead of projects with uncertain probabilities of success, a network defender faces network nodes that can be attacked at uncertain Poisson-distributed rates. This new technique, which by similarity we call "multi-node bandits, " takes a dynamic approach to cybersecurity investment, exploring how network defenders can optimally allocate cyber defense teams among nodes in their network. In effect, this strategy involves taking teams that traditionally respond to cyber breaches after they occur, and instead employing them in a proactive manner for defensive and information gathering purposes. We apply this model to a case study of an electric utility considering the degree to which to integrate demand response technology into their smart grid network, jointly identifying both the optimal level of connectivity and the optimal strategy for the sequential allocation of cybersecurity resources. Additional analytical and empirical results demonstrate the extension of the model to handling a range of practical network security applications, including sensitivity analysis to organization-specific security factors, settings with dynamic or dependent rates of attack, or handling defense teams as imperfect detectors of cyberattacks.

This thesis provides an in-depth analysis of two major components in the design of loyalty reward programs. First, we discuss the design of coalition loyalty programs - schemes where customers can earn and spend reward points across multiple merchant partners. And second, we conduct a model based comparison of a standalone loyalty reward program against traditional pricing - we theoretically characterize the conditions under which it is better to run a reward program within a competitive environment. Coalition loyalty programs are agreements between merchants allowing their customers to exchange reward points from one merchant to another at agreed upon exchange rates. Such exchanges lead to transfer of liabilities between merchant partners, which need to be frequently settled using payments. We first conduct an empirical investigation of existing coalitions, and formulate an analytical model of bargaining for merchant partners to agree upon the exchange rate and payment parameters. We show that our bargaining model produces networks that are close to optimal in terms of social welfare, in addition to cohering with empirical observations. Then, we introduce a novel alternate methodology for settling the transferred liabilities between merchants participating in a coalition. Our model has three interesting properties -- it is decentralized, arbitrage-proof, and fair against market power concentration -- which make it a real alternative to how settlements happen in coalition loyalty programs. Finally, we investigate the design of an optimal reward program for a merchant competing against a traditional pricing merchant, for varying customer populations, where customers measure their utility in rational economic terms. We assume customers are either myopic or strategic, and have a prior loyalty bias toward the reward program merchant, drawn from a known distribution. We show that for the reward program to perform better, it is necessary for a minimum fraction of the customer population to be strategic, and the loyalty bias distribution to be within an optimal range. This thesis is a useful read for marketers building promotional schemes within retail, researchers in the field of marketing and behavioral science, and companies investigating the intersection of customer behavior, loyalty, and virtual currencies.

How can entrepreneurial firms in emerging economies effectively innovate, grow, and achieve high performance? In this dissertation, I conduct a detailed examination of the institutional and relational challenges that entrepreneurial ventures in emerging economies face as well as strategies they may utilize to overcome these challenges. Three distinct papers constitute the core of this cumulative dissertation. In the first paper, I examine how inconsistencies in the institutional environment affect entrepreneurial ventures' ability to innovate and achieve high performance. In the second paper, I examine how entrepreneurial ventures may establish new organizational roles and responsibilities in spite of contradictory roles and hierarchical positions in the social lives of the entrepreneurs that founded these ventures. In the third paper, I examine how the social relationships of venture founders may sometimes give rise to malfeasance—and how these founders may curb this distinct form of malfeasance. Taken together, I examine the institutional and relational challenges that entrepreneurial ventures in emerging economies face while also presenting insights on how such challenges may be overcome.

Online crowdsourcing marketplaces provide access to millions of individuals with a range of expertise and experiences. To date, however, most research has focused on microtask platforms, such as Amazon Mechanical Turk. While microtask platforms have enabled non-expert workers to complete goals like text shortening and image labeling, highly complex and interdependent goals, such as web development and design, remain out of reach. Goals of this nature require deep knowledge of the subject matter and cannot be decomposed into independent microtasks for anyone to complete. This thesis shifts away from paid microtask work and introduces diverse expert crowds as a core component of crowdsourcing systems. Specifically, this thesis introduces and evaluates two generalizable approaches for crowdsourcing complex work with experts. The first approach, flash teams, is a framework for dynamically assembling and computationally managing crowdsourced expert teams. The second approach, flash organizations, is a framework for creating rapidly assembled and reconfigurable organizations composed of large groups of expert crowd workers. Both of these approaches for interdependent expert crowd work are manifested in Foundry, which is a computational platform we have built for authoring and managing teams of expert crowd workers. Taken together, this thesis envisions a future of work in which digitally networked teams and organizations dynamically assemble from a globally distributed online workforce and computationally orchestrate their efforts to accomplish complex work.

In this work, I conceptualize innovation as a core institution of the world economy and society, and study the factors involved in its expansion across the globe. Specifically, I examine local conditions and global institutional pressures affecting a country's willingness to take part in the innovation economy, its level of expenditure on innovation, and the actual product of those expenditures. I use event-history models and panel regressions to test the aforementioned relationships on a sample of 132 countries from 1996 to 2012. There are four main findings. First, developing nations that are close to the advanced countries (in the core of the global political economy) are likely to follow the norms of the core and initiate actions to be considered as participants of the innovation economy. Second, countries are likely to mimic neighbors in their innovation spending patterns. Third, most developing regions of the world show signs of decoupling (producing significantly less innovation patents at the same level of R& D spending than the core). And, fourth, local competition is associated to mimetic isomorphism regarding innovation but also to lower levels of decoupling in the production of patents (by increasing the efficiency of R& D investments). These results suggest that there is imitation of the global core and of neighbors (oftentimes) despite sub-standard levels of efficiency, and that participating in innovation is as important as producing actual innovation.

Mathematical models of the energy system are very successful tools for domestic policy analysis. However, these models have not been widely used to support decision making at the intersection of energy and U.S. foreign policy. This dissertation argues energy modeling must adapted to be relevant in this unique context including making greater use of uncertainty analysis. Criteria are identified to contrast four approaches to uncertainty analysis: predictive scenario analysis, Monte Carlo analysis, decision analysis, and exploratory modeling and analysis. Using an optimization and simulation model of the energy system, each approach to uncertainty analysis is used to analyze a current U.S. foreign policy problem: Should the United States provide incentives to promote natural gas in the electricity mix of low income countries? The results of the analysis are used to make a recommendation about U.S. policy and about the approach to uncertainty analysis that is most appropriate for energy and foreign policy decision making. Based on the approaches contrasted, decision analysis and exploratory modeling and analysis used together are found to have the greatest potential as a tool for energy and foreign policy analysis. When combined, these approaches have the most suitable implications for encoding uncertain variables, conducting broad policy search, quickly providing updated results, producing results that can be correctly interpreted by decision makers to provide direction and intuition, and directing the collection and integration of new information. In combination these approaches shift the burden of probabilistic reasoning away from the decision maker, while providing insights into the energy system that a decision maker can integrate with other types of policy judgment.

Decision makers in government, industry, and academia need to consider adversaries with harmful intentions when planning the development and adoption of some new technologies. Typical decision models for technology focus on commercial applications where patent law or market economics strongly influence competition. Existing models do not address all cases where adversaries can co-opt or misuse even a tightly regulated technology. This dissertation extends the technology adoption literature to consider technology decisions with adversaries in conflict scenarios where technology determines outcomes for the decision makers. We develop a general framework and model that is intended to capture strategic issues in technology decision making using a Markov game. Then we use the model to answer contemporary risk management questions in two diverse fields: cyber operations (a high-obsolescence technology field) and synthetic biology (a case study on dual-use influenza viral research). Technology R& D, conflict events, and external random events are modeled as stochastic processes for each player. These cases seem unrelated—but they share mathematical features that benefit from analysis with our model.

Prior work suggests that when organizations are faced with multiple performance demands, performance is diminished unless it is concentrated on only one set of those demands. However, recent scholarship on "hybrid" organizations describes organizations that arrange to simultaneously fulfill multiple types of demands. Do hybrid organizations tend to achieve comprehensive performance on more than one demand at once? In this dissertation study, I use quantitative and qualitative methods to address the question of hybrid organizational performance in the context of microfinance. Microfinance is an industry where small, "micro" loans are given to borrowers in extremely low income situations, often with the intent to lift the recipient out of poverty. For qualitative study, I review secondary resources and perform numerous informal interviews with industry affiliates, complete semi-structured interviews with 38 industry affiliates, and analyze keyword frequency patterns in 1,339 organizational mission statements. For quantitative analysis, I compare the performance of non-hybrid and hybrid organizations relative to two types of demands (social and financial), by analyzing 10,069 firm years of data from 1,640 organizations in the microfinance industry over the past fifteen years. In this industry, two types of hybrid and two types of non-hybrid organizations regularly report both their social and financial performance. These industry characteristics allow for testing hypotheses related to hybrid organizational performance and for investigation of a key contingency. I develop a theoretical model which proposes that when a hybrid organization combining two logics is dominated by a logic that is negatively valenced (i.e., repulsed by) its counterpart logics, then hybrid organizational performance will tend to be diminished. When a hybrid organization is dominated by a logic that is positively valenced (i.e., attracted to) its counterpart logic, then the organization will tend to achieve more comprehensive performance. Empirical study of informal interviews, semi-structured interviews, mission statements, and annual performance data largely corroborate the theoretical model. This study's findings have implications for hybrid organizations combining multiple logics, for organizations reconciling social and financial performance, and for organizations dealing with various other combinations of multiple, concurrent demands.

Significant uncertainty surrounds cyber security investments. Chief information officers (CIOs) operate with limited resources and typically do not know the relative risk of different cyber attack vectors, such as malicious email, website attacks, or lost laptops. Therefore, CIOs currently have difficulty in assessing the risk reduction associated with different cyber security investments, possibly resulting in a poor allocation of resources. For example, an organization might dedicate significant resources to detecting malicious insiders, even though its risk from website hacking is much larger. Presently, cyber risk is managed qualitatively in most organizations. Current best practices rarely incorporate quantitative risk tools and instead largely advocate the use of risk matrices, which are ambiguous and lack the ability to incorporate system dependencies. This dissertation discusses the application of probabilistic risk analysis (PRA) to cyber systems, which allows decision makers to rigorously assess the value of cyber security safeguards. First, different classes of attack scenarios are modeled. For example, laptops are lost or stolen, websites are defaced, phishing emails attempt to steal employee credentials, and malware infects machines via web browsing. Next, the rate and consequences of each scenario are assessed, drawing heavily from historical data at organizations, academic literature, publicly available data, and expert knowledge. In the case of large or rare cyber incidents where sufficient data do not exist, scenario analysis is used to obtain probabilistic assessments. These data initialize a Monte Carlo simulation to calculate probability distributions of monetary losses resulting from cyber incidents at the organization. Next, safeguards are considered that change the rate or impact of the scenarios. Changing the model structure or the model inputs shows how each safeguard affects the consequence distribution, essentially demonstrating the value of each safeguard. Sensitivity analysis can also be performed to identify the important uncertainties and the robustness of different safeguard implementation decisions. The process described above is a framework for the quantitative assessment of cyber risk in dollar terms. The result is that cyber security safeguards can be valued and prioritized. To demonstrate this framework in action, this dissertation describes a general model combined with a detailed case study of cyber risk quantification at a large organization. Over 60,000 cyber security incidents from that organization are analyzed and used to initialize the model to determine the cost-effectiveness of security safeguards including full disk encryption, two-factor authentication, and network segmentation. These data provide useful statistics for low and medium level incidents, but some incidents may be absent from the data because large incidents have not yet occurred, or have occurred too rarely to obtain good estimates for the probabilities. In this case, classes of scenarios are modeled and initialized with conditional probabilities elicited from experts. The data driven model is combined with the scenario based model by overlapping the two cost curves to ensure that incidents are not double counted, resulting in a complete and comprehensive assessment of cyber risk at the organization. Risk quantification is a critical requirement for organizations. A lack of real-world data and massive uncertainty about cyber impacts has limited progress, but organizations can now be armed with the information and tools needed to measure cyber risk. Cyber security continues to be a rapidly evolving domain, but risk quantification illuminates the cyber landscape and enables defenders to improve resource allocation and optimize decision making.

Stochastic models of event timing are popular in many applications because of their ability to capture random event arrivals and their impacts. For the particular settings of point processes with stochastic intensities, this dissertation develops a simulation and estimation methodologies. Specifically, this dissertation constructs efficient importance sampling estimators of certain rare-event probabilities involving affine point processes under a large pool regime. The proposed computational approach is based on dynamic importance sampling, and the design of the estimators extends past literature to accommodate the point process settings. In particular, the state-dependent change of measure is performed not at event arrivals but over a deterministic time grid. Several common criteria for optimality of the estimators under limited computational resources are analyzed. Numerical results illustrate the advantages of the proposed estimators. Additionally, this dissertation establishes accurate estimators of bond illiquidity from bond transaction records. The measure of bond illiquidity is defined as the sum of a round-trip transaction cost, i.e., a bid-ask spread, and a carrying cost. The carrying cost is the compound cost monetizing the waiting time for counterparties while locked in or holding a bond. Transaction times and their bid-ask spreads are modeled by arrivals and marks of a marked point process so that bond illiquidity is estimated by an effective bid-ask spread and a certain function of its intensity. Numerical results illustrate the advantages of the proposed measure and volume-related trends.

In this thesis, I develop a deep learning model for predicting house prices in the U.S. House price prediction has been increasingly important to many sectors of the economy, such as real estate, mortgage investment and risk management, and government property tax collection. Among all the factors, foreclosure is believed to have a huge impact on house prices. When the real estate bubble burst after the 2008 subprime crisis, there has been a significant portion of sales being foreclosed. This motivates researchers to investigate how foreclosure impacts house prices and by which channels it happens. Here I use an unprecedented dataset provided by the data vendor CoreLogic of over 100 million housing transactions across the U.S. from 2004 to 2014. Within my deep learning framework, I handle missing data through an efficient imputation method, address spatial and temporal effects through new features created, and finally achieve a superior out-of-sample predictive performance on datasets across the country. With the fitted model, I further explore the relationship between foreclosure discount and other variables such as the underlying house price, the age of the house and the neighborhood house price, which couldn't be fully characterized by previous linear models. I find significant regional differences as well as universal patterns across different areas.

There is a strong consensus that the climate is changing, that human activities are the dominant cause of this change, and that continued climate change will have negative impacts on human societies. To analyze energy and climate policy remedies, researchers have developed a diverse collection of integrated assessment models (IAMs) that represent the linked energy, economic, and earth systems in an interdisciplinary framework. Some IAMs are cost-benefit models designed to compute optimal policy interventions, while others are cost-effectiveness models used to determine the technology pathways that enable an emissions or climate goal to be achieved at least cost. Although IAM representations of technological change are critical determinants of model outcomes, underlying processes are poorly understood and models typically feature fairly crude formulations. The goal of the three projects that constitute this dissertation is to develop more advanced representations of technological change that capture a wider range of endogenous drivers. Scenario analyses based on these representations reveal their implications for energy and climate policy, as well as technology transitions this century. Chapter 2 describes the development of a system of technology diffusion constraints that endogenously respects empirically observed spatial diffusion patterns. Technologies diffuse from an advanced core to less technologically adept regions, with adoption experiences in the former determining adoption possibilities in the latter. Endogenous diffusion constraints are incorporated into the MESSAGE framework and results suggest that IAMs based on standard exogenous diffusion formulations are overly optimistic about technology leapfrogging potential in developing countries. Findings also demonstrate that policies which stimulate initial deployment of low-carbon technologies in advanced economies can be justified from a global common goods perspective even if they fail the cost-benefit test domestically. In Chapter 3, learning-by-doing is formulated as a firm-level rather than an industry-level phenomenon. Wind and solar PV manufacturers strategically choose output levels in an oligopoly game with learning and inter-firm spillovers. This game-theoretic representation of renewable technology markets is coupled to MESSAGE so that the energy system planner can only invest in wind and solar PV capacity at the equilibrium prices the market would charge for the desired quantities. Findings illustrate that the most ambitious emissions reduction pathways include widespread solar PV diffusion, which only occurs if competitive markets and spillovers combine to reduce prices sufficiently. The relationship between price and cumulative capacity is similar to that between unit cost and cumulative capacity under competitive markets, but a combination of market power, strong climate policy, and weak spillovers can cause prices to rise with cumulative capacity even though unit costs decline. The bilevel modeling framework of Chapter 4 is built to determine the optimal combination of technology-push and demand-pull subsidies for a given technology policy application. Firms (inner agents) solve a two-stage stochastic profit maximization problem in which they choose process and product R& D investments in the first stage, then choose output levels in the second stage. The policymaker (outer agent) seeks to identify the combination of policies that induces the firms to reach an equilibrium with the highest possible expected welfare. Numerical simulation results show that technology policy can enhance welfare under a wide range of parameter settings. Spillovers reduce product R& D expenditures but generally improve welfare by making R& D more effective. Welfare decreases with competition in the no-policy case, but increases with competition if optimal technology policies can be imposed. Each of the three projects focuses on a distinct aspect of technological change, but the formulations developed for these studies reflect several important themes: endogenous mechanisms, multiple decision-making agents, game-theoretic interactions, market power, spillovers, regional heterogeneity, and uncertainty. While the research presented in this dissertation advances the modeling of technological change, a number of formidable challenges remain. The final chapter discusses some of these challenges and ideas for future research to address them.

How do low-power actors compete in intermediated markets? Prior research on intermediated markets has often taken the perspective of powerful intermediaries. In this dissertation, I take the perspective of low-power actors and examine how they compete. Three distinct papers constitute the core of this cumulative dissertation. In the first paper, a conceptual analysis, I examine how complementors manage their dependence on platform owners in platform-based markets. In the second paper, I examine how sellers resolve the "big fish, big pond" dilemma in their choice of intermediaries. In the third paper, I conduct an empirical analysis of how competition in the intermediary portfolio affects the success of employees' intra- and entrepreneurial activities. Taken together, I examine the antecedents and consequences of the strategies that low-powered actors can use to compete in intermediated markets.

Tuberculosis (TB) is a complex infectious disease that kills millions of people every year despite effective existing treatments. Even worse, drug-resistant (DR) and multidrug-resistant (MDR) forms of TB are becoming more predominant. How should resource-constrained countries implement policies to combat TB, particularly as new diagnostic technology becomes available? This dissertation focuses on examining questions surrounding drug resistance and technology adoption in the context of the Indian TB epidemic. With one of the largest TB burdens in the world and a limited TB budget, India is an important front in the battle to control TB. In Chapter 2, we examine the relationship between treatment-generated and transmission-generated DR TB, and we evaluate the potential effectiveness of the following two disease control strategies in reducing the prevalence of DR TB: a) improving treatment of non-DR TB; b) shortening the infectious period between the activation of DR TB and initiation of effective DR treatment. We develop a dynamic transmission microsimulation model of TB in India. The model follows individuals by age, sex, TB status, drug resistance status, and treatment status and is calibrated to Indian demographic and epidemiological TB time trends. We find that the proportion of transmission-generated DR TB will continue rise over time. Strategies that disrupt DR TB transmission are projected to provide greater reductions in DR TB prevalence compared with improving non-DR treatment quality. Therefore as transmission-generated DR TB becomes a larger driver of the DR TB epidemic in India, rapid and accurate DR TB diagnosis and treatment will become increasingly effective in reducing DR TB cases compared to non-DR TB treatment improvements. Policies that use new rapid diagnostics may interrupt the transmission pathway, but their effectiveness may be undercut by inaccurate diagnosis and care inaccessibility in India. We evaluate the cost-effectiveness of policies that use rapid diagnostics, transfer patients to clinics that use WHO-approved TB treatment regimens, or combinations of these policies in Chapter 3. We extend the microsimulation model developed in Chapter 2 and additionally evaluate lifetime costs and health outcomes for each of the policies considered. We find that both types of policies (rapid diagnosis and transferring patients) and combination of policies improve health and increase costs relative to the status quo, and all but the rapid diagnosis policies alone would be cost-effective according to WHO thresholds for cost-effectiveness. While combination policies would garner the most health benefits, they would also cost the most, and our results suggest that if budget constraints necessitate implementing one before the other, programs that increase the number of patients going to clinics using WHO-approved TB treatment regimens should be prioritized over rapid diagnosis policies. In Chapter 4, we design a partially observed Markov decision process (POMDP) model for determining when rapid diagnostics should be used for drug sensitivity testing (DST) in first-line TB treatment in India, should it be adopted nationwide. We present structural properties of the model and analytical results. We find that the optimal timing of DST is influenced by availability of TB test results, level of TB transmission, and prevalence of DR TB. We find that India should revise the testing protocol in its first-line national TB treatment program to provide DST during the first month of treatment in areas of average or high DR TB prevalence and transmission. In regions with low DR TB prevalence and transmission, individually tailored testing regimens can reduce cost while maintaining health benefits of treatment. Hundreds of thousands of patients begin first-line TB treatment in India each year. We estimate that using an improved testing protocol for one year could save India up to $2.5 billion by preventing downstream transmission. In this dissertation, we present both practical and methodological contributions in the area of health policy modeling. The methods we develop can be adapted to other diseases and settings and can be useful for informing other policy decisions surrounding the control of infectious disease in resource-constrained settings.

The "Big Data" revolution is spawning systems designed to make decisions from data. Statistics and machine learning has made great strides in prediction and estimation from any fixed dataset. However, if you want to learn to take actions where your choices can affect both the underlying system and the data you observe, you need reinforcement learning. Reinforcement learning builds upon learning from datasets, but also addresses the issues of partial feedback and long term consequences. In a reinforcement learning problem the decisions you make may affect the data you get, and even alter the underlying system for future timesteps. Statistically efficient reinforcement learning requires "deep exploration" or the ability to plan to learn. Previous approaches to deep exploration have not been computationally tractable beyond small scale problems. For this reason, most practical implementations use statistically inefficient methods for exploration such as epsilon-greedy dithering, which can lead to exponentially slower learning. In this dissertation we present an alternative approach to deep exploration through the use of randomized value functions. Our work is inspired by the Thompson sampling heuristic for multi-armed bandits which suggests, at a high level, to "randomly select a policy according to the probability that it is optimal". We provide insight into why this algorithm can be simultaneously more statistically efficient and more computationally efficient than existing approaches. We leverage these insights to establish several state of the art theoretical results and performance guarantees. Importantly, and unlike previous approaches to deep exploration, this approach also scales gracefully to complex domains with generalization. We complement our analysis with extensive empirical experiments; these include several didactic examples as well as a recommendation system, Tetris, and Atari 2600 games.

Planning for treatment of cancer is challenging due to the complexity of the disease. Oncologists have to select and switch therapies considering the trade-off between treatment efficacy and therapy sides effects for an increasing number of therapies and combinations. Moreover, cancer varies with different patient characteristics and past treatment history. We develop a framework for computing a dynamic strategy for therapy choice in a large class of breast cancer patients, as an example of approaches to personalize therapies for individual characteristics and each patient's response to therapy. Our model maintains a Markov belief about the effectiveness of the different therapies and updates it as therapies are administered and tumor images are observed. We compare three different approximate methods to solve our analytical model against standard medical practice and show significant potential benefit of the computed dynamic strategies to limit tumor growth and to reduce the number of time periods patients are given chemotherapy, with its attendant side effects. We test robustness of our model with sensitivity analysis on key model parameters, seeing how the optimal dynamic strategy changes when patient characteristics differ. We also demonstrate the scalability of our model and recommended algorithm, showing that it has potential of providing real time advice to patients and oncologists.