## Section: New Results

### Decentralized Learning

**Trade-offs in Large-Scale Distributed Tuplewise Estimation and Learning**
The development of cluster computing frameworks has allowed
practitioners to scale out various statistical estimation and
machine learning algorithms with minimal programming effort. This is
especially true for machine learning problems whose objective
function is nicely separable across individual data points, such as
classification and regression. In contrast, statistical learning
tasks involving pairs (or more generally tuples) of data points-such
as metric learning, clustering or ranking-do not lend themselves as
easily to data-parallelism and in-memory computing.
In [13], we investigate how to balance between
statistical performance and computational efficiency in such
distributed tuplewise statistical problems. We first propose a
simple strategy based on occasionally repartitioning data across
workers between parallel computation stages, where the number of
repartition-ing steps rules the trade-off between accuracy and
runtime. We then present some theoretical results highlighting the
benefits brought by the proposed method in terms of variance
reduction, and extend our results to design distributed stochastic
gradient descent algorithms for tuplewise empirical risk
minimization. Our results are supported by numerical experiments in
pairwise statistical estimation and learning on synthetic and
real-world datasets.

**Who started this rumor? Quantifying the natural differential privacy guarantees of gossip protocols**
Gossip protocols, also called rumor spreading or epidemic protocols,
are widely used to disseminate information in massive peer-to-peer
networks. These protocols are often claimed to guarantee privacy
because of the uncertainty they introduce on the node that started
the dissemination. But is that claim really true? Can one indeed
start a gossip and safely hide in the crowd?
In [14], we study gossip protocols using a
rigorous mathematical framework based on differential privacy to
determine the extent to which the source of a gossip can be
traceable. Considering the case of a complete graph in which a
subset of the nodes are curious, we derive matching lower and upper
bounds on differential privacy showing that some gossip protocols
achieve strong privacy guarantees. Our results further reveal an
interesting tension between privacy and dissemination speed: the
standard “push” gossip protocol has very weak privacy guarantees,
while the optimal guarantees are attained at the cost of a drastic
increase in the spreading time. Yet, we show that it is possible to
leverage the inherent randomness and partial observability of gossip
protocols to achieve both fast dissemination speed and near-optimal
privacy.

**Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs**
In [15], we consider the fully
decentralized machine learning scenario where many users with
personal datasets collaborate to learn models through local
peer-to-peer exchanges, without a central coordinator. We propose
to train personalized models that leverage a collaboration graph
describing the relationships between the users' personal tasks,
which we learn jointly with the models. Our fully decentralized
optimization procedure alternates between training nonlinear models
given the graph in a greedy boosting manner, and updating the
collaboration graph (with controlled sparsity) given the
models. Throughout the process, users exchange messages only with a
small number of peers (their direct neighbors in the graph and a few
random users), ensuring that the procedure naturally scales to large
numbers of users. We analyze the convergence rate, memory and
communication complexity of our approach, and demonstrate its
benefits compared to competing techniques on synthetic and real
datasets.

**Advances and Open Problems in Federated Learning**
Federated learning (FL) is a machine learning setting where many clients
(e.g. mobile devices or whole organizations) collaboratively train a model
under the orchestration of a central server (e.g. service provider), while
keeping the training data decentralized. FL embodies the principles of
focused data collection and minimization, and can mitigate many of the
systemic privacy risks and costs resulting from traditional, centralized
machine learning and data science approaches. Motivated by the explosive
growth in FL research, we participated in a collaborative paper [18]
that discusses recent advances and presents an extensive collection of open
problems and challenges.