Running ROS 2 on Multiple Machines
ROS enables the operation of nodes on an individual robot or across multiple robots. In this guide, you'll discover how to set up ROS to function on several computers and robots. We'll explore two scenarios: when robots are within a local network and when they're connected via the Internet.
Prerequisitesβ
To engage with the guides in this tutorial, you simply need two devices running ROS 2. One can be your personal laptop, while the other could be either a ROSbot XL, ROSbot 2R or ROSbot 2 PRO, all of which come with ROS 2 Humble pre-installed. It is also possible to solve the tutorial using the Gazebo simulation.
For the tutorial on Internet connections, we utilize the Husarnet VPN. While it streamlines and automates a significant portion of the setup, the functionalities of Husarnet highlighted in this context primarily center around facilitating ROS 2 communication over the Internet. However, this doesn't encompass all that Husarnet offers. Dive deeper into its capabilities by exploring the official Husarnet documentation and blog.
If you don't already own a ROSbot, you can get one from the Husarion store.
The tutorials are designed to flow sequentially and are best followed in order. If you're looking to quickly assess the content of a specific tutorial, refer to the ROS 2 Tutorials repository, which contains the completed outcomes.
Connecting ROS 2 in LANβ
The default settings in ROS 2 ensure that all devices within the same subnet are connected, provided that multicasting is activated. ROS 2 operates on a layer known as the "ROS Middleware" (RMW). Multiple RMW implementations exist, all based on the DDS (Data Distribution Service) connectivity pub-sub framework. This framework ensures seamless communication between various components of the ROS 2 nodes running in the same subnet.
1. Connect devices to the LAN
The first thing you need to do is connect them to the same LAN. You can check the address of your device with the command.
hostname -I
Then you can verify the connections. On second device type:
ping <ROSBOT_IP>
2. Run example
Now, using the ROS 2 CLI, post a custom message of type String
to the /msg
topic every 1 second.
ros2 topic pub -r 1 /msg std_msgs/msg/String data:\ 'Hello, ROSbot here'
ros2 topic echo /msg
You should see a greeting. It's amazing how quickly the result was achieved. π
While the default information exchange among all connected devices is convenient, it does come with a significant drawback: What if there's another robot on the network with which we would prefer not to share data? Fortunately, the ROS 2 developers anticipated this issue and have provided a straightforward solution.
Using ROS_DOMAIN_ID
β
To ensure that only specific robot components communicate with each other on a shared network, DDS utilizes a unique code known as a "domain identifier." This mechanism manages communication and prevents potential miscommunications. To modify the domain ID, you need to set the ROS_DOMAIN_ID
environment variable.
- First, set the
ROS_DOMAIN_ID
environment variable on the first device and start transmitting messages.
export ROS_DOMAIN_ID=1
ros2 topic pub -r 1 /msg std_msgs/msg/String data:\ 'Hello, ROSbot here'
- Next, check whether the laptop receives the message. Because of the domain code adjustment, the other device shouldn't intercept any messages now.
ros2 topic list
- Finally, set the
ROS_DOMAIN_ID
on the second device to the same value, and then retrieve the messages.
export ROS_DOMAIN_ID=1
ros2 topic echo /msg
- You can safely set the
ROS_DOMAIN_ID
value in the range of0
to101
, inclusive. For more details, click here. - Keep in mind that configuring
ROS_DOMAIN_ID
is session-specific. If you launch a new terminal, you'll need to configure this variable once again.
Using ROS_DISCOVERY_SERVER
β
Another DDS mechanism can segregate 'robot networks' within the same LAN. Instead of depending on the standard multicast-based LAN discovery, which is the default for DDS, you have the option to transition to the Discovery Server approach for DDS discovery. It's important to note that this feature is currently exclusive to FastDDS. FastDDS serves as the default RMW for ROS 2 versions like Foxy (LTS, 2020), Humble (LTS, 2022), and Iron (2023).
- First, run the discovery server on one of the devices.
fastdds discovery --server-id 0 --port 11888
- Link both the ROSbot and the laptop to the same discovery server. Since the server is initiated on the ROSbot, its LAN IP address is essential. Given that the ROSbot's IP address is
10.5.10.130
, it is necessary to configure theROS_DISCOVERY_SERVER
environment variable on each device. This ensures that ROS 2 nodes utilize the Discovery Server instead of the multicast-based discovery method.
export ROS_DISCOVERY_SERVER="10.5.10.130:11888"
ros2 daemon stop # reload ROS 2 daemon
ros2 run demo_nodes_cpp talker
export ROS_DISCOVERY_SERVER="10.5.10.130:11888"
ros2 daemon stop # reload ROS 2 daemon
ros2 run demo_nodes_cpp listener
After you set the ROS_DISCOVERY_SERVER
, you will notice that only two default topics are available.
/parameter_events
/rosout
From a user experience standpoint, it might seem like a flaw. However, when viewed from a network traffic perspective, it's an advantage: discovery traffic is only exchanged among participants who have common topics, either as publishers or subscribers. To view all ROS 2 topics across hosts connected to the same Discovery Server, you should opt for the SUPER_CLIENT
configuration instead of using ROS_DISCOVERY_SERVER
, which sets up the CLIENT
configuration. To set up the SUPER_CLIENT
config:
- Create the
fastdds_supeclient.xml
file with the following content:
Show me fastdds_supeclient.xml
<?xml version="1.0" encoding="UTF-8"?>
<dds>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
<participant profile_name="super_client_profile" is_default_profile="true">
<rtps>
<builtin>
<discovery_config>
<discoveryProtocol>SUPER_CLIENT</discoveryProtocol>
<discoveryServersList>
<RemoteServer prefix="44.53.00.5f.45.50.52.4f.53.49.4d.41">
<metatrafficUnicastLocatorList>
<locator>
<udpv4>
<address>10.5.10.130</address>
<port>11888</port>
</udpv4>
</locator>
</metatrafficUnicastLocatorList>
</RemoteServer>
</discoveryServersList>
</discovery_config>
</builtin>
</rtps>
</participant>
</profiles>
</dds>
- Setup the FastDDS default profile file environment variable to the path where the file is located:
export FASTRTPS_DEFAULT_PROFILES_FILE=$(pwd)/fastdds_supeclient.xml
ros2 topic list
# you should see all the topics
ros2 run demo_nodes_cpp listener
# ...and see messages published by the talker
Connecting ROS 2 via Internetβ
As highlighted earlier, you can operate ROS nodes across numerous devices as long as they share the same LAN network. This enables communication between the ROS on your laptop and the robot when both are linked to the same Wi-Fi router. However, the aforementioned setup won't function if the robot is connected to the Internet through a different medium, such as an LTE module.
To address the connectivity issue, you could consider leveraging a VPN service. However, it's worth noting that conventional VPNs are not primarily designed for mobile robotics and can introduce certain challenges to your setup, including:
- The need to manage a VPN server.
- Encountering higher latency, as all traffic must route through the VPN server.
- Extended reconfiguration times, especially when your robot switches between Wi-Fi hotspots.
- The manual configuration requirement for DDS.
Husarnet is a specialized VPN service tailored for ROS 2. As an open-source peer-to-peer VPN, Husarnet facilitates direct robot connections over the Internet without the need for a centralized VPN server to relay data. The traffic is safeguarded with ephemeral encryption keys, and it's seamlessly compatible with ROS. Just set up the Husarnet Client on both your laptop and robot, link them via the same Husarnet network through the user-friendly web dashboard, and enjoy the benefits of swift, direct device-to-device communication.
To set up a free Husarnet account and connect your ROS 2-powered devices, simply follow these 5 easy steps:
- 1. Create Account
- 2. Confirm e-mail
- 3. Create a Network
- 4. Copy a Join Code
- 5. Connect Devices
Create a free account at app.husarnet.com.
After you click Register
you will be redirected to your account where your will see.
Please click the confirmation link in the e-mail you should have just received:
After you click the confirmation e-mail you will see your account with no networks.
Click a Create network
button and name your first network, e.g. my_network
:
After you create a new network you will see:
Click Add element
button and you will see a window with your Join Code.
Finally, install the Husarnet Client on your devices and add them to your Husarnet network, by simply executing in the Linux terminal 3 commands:
β οΈ WARNING Husarnet is already pre-installed in ROSbot XL, ROSbot 2R and ROSbot 2 PRO
# 1. Install Husarnet
curl -s https://install.husarnet.com/install.sh | sudo bash
# 2. Connect to the VPN network using the code you obtained in step 4/5 (<your-join-code>)
sudo husarnet join <your-join-code> my-robot
# 3. Select your system architecture and install Husarnet-DDS
RELEASE="v1.3.5"
ARCH="amd64"
sudo curl -L https://github.com/husarnet/husarnet-dds/releases/download/$RELEASE/husarnet-dds-linux-$ARCH -o /usr/local/bin/husarnet-dds
sudo chmod +x /usr/local/bin/husarnet-dds
The last think is selecting appropriate DDS configuration.
- Fast DDS (Simple Discovery)
- Fast DDS (Discovery Server)
- Cyclone DDS
# 4. Configure Husarnet DDS
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
export FASTRTPS_DEFAULT_PROFILES_FILE=/var/tmp/husarnet-fastdds-simple.xml
husarnet-dds singleshot
For devices acting as a server with the Husarnet hostname my-laptop
used in the client configuration below:
# 4. Configure Husarnet DDS on server
export DISCOVERY_SERVER_PORT=11811
husarnet-dds singleshot
fast-discovery-server -i 0 -x /var/tmp/husarnet-dds/fastdds-ds-server.xml
For devices acting as a client:
# 4. Configure Husarnet DDS on client
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
export FASTRTPS_DEFAULT_PROFILES_FILE=/var/tmp/husarnet-fastdds-ds-client.xml
export ROS_DISCOVERY_SERVER=my-laptop:11811
husarnet-dds singleshot
# 4. Configure Husarnet DDS
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
export CYCLONEDDS_URI=file:///var/tmp/husarnet-cyclone.xml
husarnet-dds singleshot
Install Husarnet in the same way for the second device and configure it. Remember to change the name of your device (sudo husarnet join <your-join-code> my-laptop
). Now we can communicate with devices as if they were connected to the same local network, even if they are located on different continents.
At this point you should have your robot and laptop connected to the same Husarnet network with the following Husarnet Hostnames.
- robot:
my-robot
- laptop:
my-laptop
Manual Configuration (optional)
If you require a configuration different from the default one supplied by the Husarnet-DDS utility, you will need to generate a custom XML configuration file. The following steps outline how to create this file for both Fast DDS and Cyclone DDS.
- Fast DDS
- Cyclone DDS
Installation
If the default DDS implementation used in you ROS 2 distribution is different than FastDDS, you need to install it:
sudo apt-get update
sudo apt-get install ros-$ROS_DISTRO-rmw-fastrtps-cpp
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
Preparing a Configuration
Create a Fast DDS configuration file in your system: ~/fastdds_husarnet.xml
. To make the communication work, you have to set some parameters as shown in the following template.
<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
<transport_descriptors>
<transport_descriptor>
<transport_id>HusarnetTransport</transport_id>
<type>UDPv6</type>
</transport_descriptor>
</transport_descriptors>
<participant profile_name="CustomHusarnetParticipant" is_default_profile="true">
<rtps>
<userTransports>
<transport_id>HusarnetTransport</transport_id>
</userTransports>
<useBuiltinTransports>true</useBuiltinTransports>
<builtin>
<initialPeersList>
<!-- Repeat this part for each husernet peer -->
<locator>
<udpv6>
<address>husarnet-IPv6-address</address> <!-- or <address>husarnet-hostname</address> -->
</udpv6>
</locator>
<!-- End repeat -->
</initialPeersList>
</builtin>
</rtps>
</participant>
</profiles>
IMPORTANT: Provide all peers under
initialPeersList
tag.
Launching the Configuration
Set this file as default profile in your .bashrc
file, so as to use this configuration every time you boot your system. Open a new terminal and execute:
echo "export RMW_IMPLEMENTATION=rmw_fastrtps_cpp" >> ~/.bashrc
echo "export FASTRTPS_DEFAULT_PROFILES_FILE=/home/$USER/fastdds_husarnet.xml" >> ~/.bashrc
. ~/.bashrc
Installation
If the default DDS implementation used in you ROS 2 distribution is different than Cyclone, you need to install it:
sudo apt-get update
sudo apt-get install ros-$ROS_DISTRO-rmw-cyclonedds-cpp
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
Preparing a Configuration
Create communication settings file, for example in your home directory ~/cyclonedds.xml
. To make communication work you have to set some params as follows:
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
<Domain id="any">
<General>
<NetworkInterfaceAddress>hnet0</NetworkInterfaceAddress>
<AllowMulticast>false</AllowMulticast>
<MaxMessageSize>65500B</MaxMessageSize>
<FragmentSize>4000B</FragmentSize>
<Transport>udp6</Transport>
</General>
<Discovery>
<Peers>
<Peer address="[IPV6-address]"/> <!-- example: <Peer address="[fc94:dd2c:a2e6:d645:1a36:****:****:****]"/> -->
<Peer address="[hostname]"/> <!-- example: <Peer address="[my-laptop]"/> -->
</Peers>
<ParticipantIndex>auto</ParticipantIndex>
</Discovery>
<Internal>
<Watermarks>
<WhcHigh>500kB</WhcHigh>
</Watermarks>
</Internal>
<Tracing>
<Verbosity>severe</Verbosity>
<OutputFile>stdout</OutputFile>
</Tracing>
</Domain>
</CycloneDDS>
You can provide Peer as IPv6 address or hostname from Husarnet.
IMPORTANT: Provide all peers
<hostnames/IPv6 address>
, (chose one) in every machine.
Important fields:
-
NetworkInterfaceAddress
<NetworkInterfaceAddress>hnet0</NetworkInterfaceAddress>
. Force DDS to use hnet0 network interface (the one created in your host OS by Husarnet VPN). -
Multicast
<AllowMulticast>false</AllowMulticast>
. At the time of writing this file multicast is not supported by Husarnet so it's necessary to disable this. Multicast feature will be added soon. -
Transport Protocol
<Transport>udp6</Transport>
. It's necessary to use IPv6 but Cyclone doesn't allow to mix IPv4 with IPv6 so every node must communicate over IPv6. -
IPV6-address
<Peers><Peer address="[IPV6-address]"/></Peers>
. Here appropriate IP addresses should be filled, you can take this address form Husarnet App. Safe method is to provide address of local IPv6 and remote machines.
If you need information about params check cyclonedds-manual
Launching the Configuration
Add changes to .bashrc
file to use that configuration every time you boot your system. Open a new terminal and execute:
echo "export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp" >> ~/.bashrc
echo "export CYCLONEDDS_URI=file:///home/$USER/cyclonedds.xml" >> ~/.bashrc
. ~/.bashrc
Test connection
Post a message from the terminal using the familiar ros2 topic
command. For example, on one computer you can type:
ros2 topic pub -r 1 /msg std_msgs/msg/String data:\ 'Husarnet is awesome! :D'
and use ros2 topic echo /msg
to listen for incoming topic.
More about Husarnet VPNβ
Husarnet is open source and free to use up to 5 devices. Here you can find many articles articles among which you will find information describing the principle of operation of Husarnet and many examples of using Husarnet in many applications (e.g. inside Docker).
Nice! At this point you should have your laptop and robot connected via LAN or Internet. Let's run some additional demo nodes to see how the connection works in practice.
Real-time video streamingβ
In the first demo, the camera on one of the platforms will be launched. Thanks to the network connection, we can access any topic between devices. In this example, we will show you how to display the image from the camera on the second device.
- ROSbot & Laptop
- Two laptops
ROSbot
-
Configure
ROS_DOMAIN_ID
for LAN network, or use Husarnet VPN. -
Launch ROSbot:
docker compose up -d microros <rosbot_service> <camera_service>
Gazebo
-
Configure
ROS_DOMAIN_ID
for LAN network, or use Husarnet VPN. -
Launch Gazebo simulation:
ROSBOT_SIM
Running the simulator is very easy thanks to the created aliases. If you haven't done so already check configure your environment.
Laptop
-
Configure
ROS_DOMAIN_ID
for LAN network, or use Husarnet VPN. -
Check if connection works with:
ros2 topic list
- Run RViz2 to see a real-time video from the robot. Select topic and QoS policy to get image visualization.
rviz2
Reference projectsβ
Below we present a project that allows for the rapid exchange of information in the network by using image data compression, which reduces delays in image transmission by reducing the size of the data.
Project | Description |
---|---|
rosbot-telepresence rosbot-xl-telepresence | Control the robot over the Internet with real-time camera feed |
Summaryβ
And there you have it! At this point, you should have a solid understanding of how to manage robots connected via LAN or the Internet. For LAN-based setups, the primary focus is on determining the appropriate domain using ROS_DOMAIN_ID
. When dealing with Internet connections, Husarnet, paired with the Husarnet-DDS utility (which automates DDS configuration), is a viable option, or you can opt to manually integrate a VPN with DDS. It's noteworthy that DDS is versatile in its configuration options. For further insights, delve into the Quality of Service, Husarnet documentation, and Husarnet blog. Having gone through this tutorial, you should be proficient in orchestrating multi-device interactions and skilled in transmitting and retrieving data from afar, empowering you to effectively command your robot.
by RafaΕ GΓ³recki, Husarion
Need help with this article or experiencing issues with software or hardware? π€
- Feel free to share your thoughts and questions on our Community Forum. π¬
- To contact service support, please use our dedicated Issue Form. π
- Alternatively, you can also contact our support team directly at: support@husarion.com. π§