Request for help implementing HID protocol in BrailleTouch project #14829
Replies: 29 comments 35 replies
-
|
Have you read our documentation on braille HID? |
Beta Was this translation helpful? Give feedback.
-
|
We have reviewed the documentation, but we have not found any practical examples that could assist us. There is no code example for a Braille display using HID anywhere on the internet. This complicates matters because without an example, it becomes difficult, especially for those who are not familiar with HID. It would be practical to have an example code. In fact, what we are looking for is to create a library for HID Braille, similar to the libraries available for Arduino and ESP32, that converts it into a keyboard. Users can simply load the keys they want, and the library will generate the reports and all other necessary tasks. It would be ideal to have a HID Braille library that can create Braille displays of different sizes, for example, a 20-cell display that uses specific pins, and the library can create the corresponding report list. If someone could provide an example code with the main buttons connected to some of the ESP32 pins, such as the button to read the next or previous message, forward and backward arrows, and serial console output that displays the Braille content. It would be very helpful to have this example code. This would allow us to connect it to the Braille reader, display the Braille on the console, and navigate the text using buttons. This would be a good starting point to continue working with all the required functions. I have been requesting assistance for over a year and have sought help from many programmers. However, none of them have been able to provide any code. Currently, we have been struggling for almost a week, spending entire nights trying to generate code and understand the explanations, but have not been successful. |
Beta Was this translation helpful? Give feedback.
-
|
CC @LeonarddeR I'm not sure, but it is possible you might have some knowledge
that would guide this effort in some better direction.
|
Beta Was this translation helpful? Give feedback.
-
|
I'm afraid not. I have only a small amount of knowledge about the display driver side of things, definitely not about how it works in the display itself (i.e. how a display should behave like a braille HID driver). |
Beta Was this translation helpful? Give feedback.
-
|
This has been the problem we've been facing, and I've been searching for programmers, but none have been able to help because it seems that no one has seen code for HID braille on the braille display (firmware). Would it be necessary to reverse engineer a braille display? Apparently, the part that goes on the screen reader's side (the driver) is known, but no one knows how to do the firmware or the HID part that goes from the braille display side. There is no starting example. It would be a great advancement to have a sample HID braille display code in C that can display braille content on the console and have at least two buttons to navigate through the content. With this code, we could show how to pass buttons and receive and display content on the console. It would be a perfect example for developing anything, but unfortunately, it doesn't exist. I even tried using artificial intelligence to generate code, but it also failed because everyone only works on the code that goes to a controller for making a controller, and AI does the same. |
Beta Was this translation helpful? Give feedback.
-
|
cc: @FalkoBabbage @discapacidad5 which HID library did you use on your platine? Maybe these sources could help in how to design the class and the internal library on the platine to comunicate with the HiD driver in NVDA or with Windows independently. I guess you need these enhanced libraries for your platine to comunicate via HID efficiently: and this is also very useful since you can specify your own HID classes with your own enums: Does your braille display work already without any screen reader? Or is a screen reader required to use it? This treat might help further in finding some hints for functions required to be written to your HID device in order to comunicate with Windows. |
Beta Was this translation helpful? Give feedback.
-
|
"Thank you for responding, but let me tell you that I have tried all the libraries related to HID that I could find, and I have tried every possible method that I could imagine. I have also looked for programmers who could help me, but none of them have been able to provide a solution. They all give me theoretical advice on what to do, but there is a big gap between theory and practice. What I really need is someone who can apply their theory and show me an example code that actually works. The computer recognizes the device as HID, so the problem is not with the computer recognizing it. The problem is with the screen reader recognizing it correctly and sending content to it. I have managed to write some code that makes the computer recognize the device as HID, but NVDA doesn't recognize anything. Obviously, there are specific parameters that need to be placed in certain locations, and these can only be achieved with an example code that shows how to set those parameters. On the other hand, the HID examples that I have found online only show how to send key press data from the device to the computer, but there is no example on how to receive data from the computer to the device, which is the case for Braille where I need to receive the content of the Braille cells." |
Beta Was this translation helpful? Give feedback.
-
|
"In the case of the projects you mentioned, such as the Android HID project and USBD-Human-Interface, they provide HID support only for USB devices and do not work with Bluetooth devices. In my case, I am facing a problem with the ESP32, which does not have native USB support for HID Braille. I can only work with it through Bluetooth. However, as I mentioned before, the problem is not that the computer does not recognize the device as HID. The problem is that NVDA, or more specifically HID Braille, has a specific structure to define the cells and many other things, which is not the same as the structures for a keyboard or a mouse. These libraries that convert HID do not have a structure for HID Braille. Therefore, even if I manually change the descriptor and add the HID and Braille descriptor, the computer recognizes it but the screen reader does not, because there are many things that are not properly structured. That's why it is necessary to create a specific library for HID Braille or add HID Braille support to the existing library." |
Beta Was this translation helpful? Give feedback.
-
|
It seems Helptech from Germany they claim at least some of their braille devices are pure HID devices: |
Beta Was this translation helpful? Give feedback.
-
|
I guess you already used the HID conventions in this HID usage table right? |
Beta Was this translation helpful? Give feedback.
-
|
"If that table specifies how to create the HID structure, it doesn't explain how to work with it, how to send content, how to receive content, or where to load the Braille cells. There is no information on how to update or send data to the screen reader, what needs to be sent, or received. Simply specifying the structure allows the device to be recognized as HID on the computer, but the screen reader will not recognize it as HID without a proper report including information on the number and type of Braille cells (whether 8 or 6 points), and other details that are needed for the screen reader to process the information correctly. None of this information is specified in the table." |
Beta Was this translation helpful? Give feedback.
-
|
"This code using the HID structure recognizes the device as HID on the computer, but the screen reader does not recognize anything. Obviously, the HID structure is only a part of what is needed and other information is missing from the document." |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Hi, I'm sure you may have read through NVDA's own source code, but have you tried looking at HID braille standard file (source/brailleDisplayDrivers/hidBrailleStandard.py) to get a sense of how NVDA understands HID protocol? Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
"Well, to be honest, I have read all the files related to HID personally, but I couldn't understand them because it seems to explain how the reader reads the content, but there seems to be a huge difference between how it is read and how it is sent. Apparently, the structure that one sends for HID is sent in one way for the computer to process it and delivered in another way, so apparently, the structure that is sent is not the same structure that the screen reader receives. In theory, it could be said that if I understand how the screen reader reads it, I will understand how to send it, but you can try it in practice and see that it's not as simple as the theory says. I don't know much about programming, but I have searched for many people and we have all been struggling with the same problem and none of us have been able to do anything." |
Beta Was this translation helpful? Give feedback.
-
|
The big problem lies in the fact that those who provide explanations and the manufacturers never explain how to send the HiD report. They explain how to interpret it and there are instructions on how to interpret the HiD report to use it on a screen reader to create a driver, but there is no information on how to create or generate the HiD report itself, only its interpretation and structure. Sometimes I think there is a great interest in preventing anyone from understanding this, so that no one can develop a screen that can be compatible with all screen readers. I have always believed that Braille screens end up being a business. They are so expensive and there are ways to make Braille screens so affordable, but if there is no way to communicate with the screen reader, what is the point of manufacturing a screen if it cannot be functional with screen readers or if I have to wait a long time for a custom driver for a screen reader. And of course, if I want to try to bring it to paid readers like Jaws or iPhone or even Android, how long do I have to wait to see if my device driver can be recognized in their device list? The advantage of HiD is precisely that it would give that freedom to play with all this and start designing, testing and evaluating Braille screens that work without waiting for a custom driver for a screen reader. Having a working sample code would open the doors to many projects that could use HiD to offer more accessible devices to people with disabilities. Today, people with disabilities are not using Braille precisely because of the cost of the devices and only first-world countries that have the capacity or whose governments donate the devices have the ability to have a Braille screen. The rest of the people in the world do not have that possibility, and that is the vast majority. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry if I am being dummy...
The HID protocol is not only for USB devices?
If you say that
ESP32 only supports Bluetooth, you should not devellop a specific driver for your Braille display?
Rui Fontes
Às 21:16 de 15/04/2023, discapacidad5 escreveu:
… "In the case of the projects you mentioned, such as the Android HID project and USBD-Human-Interface, they provide HID support only for USB devices and do not work with Bluetooth devices. In my case, I am facing a problem with the ESP32, which does not have native USB support for HID Braille. I can only work with it through Bluetooth.
However, as I mentioned before, the problem is not that the computer does not recognize the device as HID. The problem is that NVDA, or more specifically HID Braille, has a specific structure to define the cells and many other things, which is not the same as the structures for a keyboard or a mouse. These libraries that convert HID do not have a structure for HID Braille. Therefore, even if I manually change the descriptor and add the HID and Braille descriptor, the computer recognizes it but the screen reader does not, because there are many things that are not properly structured.
That's why it is necessary to create a specific library for HID Braille or add HID Braille support to the existing library."
|
Beta Was this translation helpful? Give feedback.
-
|
"The HID protocol is a communication protocol that works on both USB and Bluetooth. In fact, that's why there are Bluetooth keyboards and mice as well as gaming devices that work via Bluetooth, and the computer recognizes them perfectly as HID devices. So, HID devices are not only for USB. In fact, the code I posted in a previous response, if you put it on an SP32, the computer will recognize it as a Bluetooth HID device." |
Beta Was this translation helpful? Give feedback.
-
|
"I think the real problem in developing HID Braille display drivers is the lack of understanding of how the HID protocol works. There isn't a deep knowledge of the protocol, which is why even very skilled programmers haven't been able to do anything with it. It would require someone who specializes in the HID protocol, and it seems that there are few people who master it or it's difficult to find someone who does. It would be interesting if some of the programmers showed more interest in this, especially since the future expects all Braille displays to exclusively use HID. The hope for the future is that even operating systems will automatically recognize a Braille display and connect it without needing a screen reader. When the HID protocol was created, it was done with that intention, just like when you connect a keyboard, whether it's Bluetooth or USB, both work with the HID protocol and are perfectly recognized by the device you connect it to. However, the population that works to develop Braille devices is very small, and in the end, the ones who end up developing direct drivers are the manufacturers. The manufacturer, as such, is not interested in having their code seen by another manufacturer. So we are in a situation where there is no example of the code currently available in open source, and my post is precisely calling for help to see if someone can do something that gives us a starting point, a code that gives us a point to build a good driver on the Braille display side." |
Beta Was this translation helpful? Give feedback.
-
|
@discapacidad5 I am totally with you, sometimes big players announce a lot of things such as standards but often they are not being thought to the end. Especially when it comes to guidance on how to achieve them. I am not total expert, but it seems you have found some how a way to design your in report and now you need an out report for the report descriptor. Is this correct? So can you already send key presses to an application in Windows without any screen reader? Anyway, I found a good tutorial on how to create a USB report descriptor, it defines in general how to create it for some usual examples like gamepads and mouse. Now we have to find examples for some code that describes a report descriptor for bluetooth devices and then for a HID braille device. And here is a tutorial which shows how to create custom HID devices with custom report descriptors in circuitpython. I could find an usage file that has been approved from Microsoft for the HID standard, I guess this is the filtered information from the table that I have already posted above. It could make it easier to find here all the relevant info without having to scroll through the whole HID conventions: @discapacidad5 did you already try the win-hid-dump tool to extract the propper HID report descriptor for the custom device? I guess this will give you an in report rather than an out report but might be worthy to try. Then you could use a report descriptor decoder such as this one to create C structures: |
Beta Was this translation helpful? Give feedback.
-
|
I found on stack overflow a code example for sending and receiving data on a custom HID device, but not sure if it works: A couple of things to note: Source: https://stackoverflow.com/questions/21606991/custom-hid-device-hid-report-descriptor |
Beta Was this translation helpful? Give feedback.
-
|
@discapacidad5 did you raise your questions on this forum ast well? I have the impression there could be some really good experts that can help further in this regard. |
Beta Was this translation helpful? Give feedback.
-
|
For anyone's reference, here is an example of a bluetooth device that uses the Gatt profile to comunicate with the host: It also describes how a report descriptor for a bluetooth device could look like. |
Beta Was this translation helpful? Give feedback.
-
|
And here is a complete report descriptor for a smart remote which might also help. |
Beta Was this translation helpful? Give feedback.
-
|
Finally, here is some more Info on how HID input and output can works between HID device and host, here is a dualshock game controller as an example:. |
Beta Was this translation helpful? Give feedback.
-
|
"Thank you for all the information you've sent, I will look into it to see if I can achieve something. However, I have seen many examples and have actually been able to make a HID device work. There are many examples of how a HID device works and there are libraries that do the job. But what I have noticed is that even if you have a HID device configured, it does not work with the screen reader. In fact, key presses on the HID Braille device do not go directly to the operating system like on a normal keyboard. The key presses are captured by the screen reader. Currently, the operating system does not recognize the HID Braille to work with it directly. That is the big problem. That's why I mentioned that it's not about just passing examples. There are many examples, but it's about making an example that works with HID Braille. When you start working on it, you realize that existing examples do not work with HID Braille because it's not enough to be recognized by the computer. It has to carry certain structures that the screen reader must recognize. All key presses have to go to the screen reader, and the screen reader looks for the desktop in a specific location. Therefore, both the input and output go to the screen reader, and the screen reader processes it. I will analyze all the examples you sent, but even those that claim to provide an example of a descriptor do so theoretically. No one has done it in practice. The theory is not enough when thinking about how HID works. Braille input is a bit different and requires certain different operating parameters. I encourage you to try to make a functional code that can receive the content of Braille and can be recognized by the screen reader, not just recognized by the operating system, as that can already be done. There are already libraries that help with that, and you simply need to load the library to be automatically recognized. But it's not enough to be recognized by the operating system. The goal is to make the screen reader recognize the HID Braille, and that's where we are falling short. There is a part of the code that is missing, something needs to be done, and there are some parameters that are not being set. Therefore, the screen reader is not recognizing it, and that's the unclear part. Everyone explains it theoretically, but no one provides a practical example. Sometimes, a practical example is necessary to understand." |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Hmmm, even after reading NVDA's own source code? If yes, then we might as well discuss how we (NVDA folks) can make our source code better so others can understand what's going on, because when discussing HID and screen readers, NVDA's own HID standard implementation is the closest we have to seeing the protocol in action. As long as a screen reader (or some assistive tech software) is running, the screen reader will process input before anyone can do so, and that also involves understanding input coming from a braille display as communicated by its driver. In case of HID, HID input is first processed by the screen reader, therefore the screen reader's job is to parse protocol messages from the braille device and translate it into actual input mechanism the operating system supports. As for actually writing support for a braille display, yes, it involves looking at or asking for basic protocol documentation, and lots of debugging to get it right (more so if one wishes to support multiple models). The last time I did something with braille input and output, I have added support for specific models of HumanWare BrailleNote family, involving different models and input styles. This work involved printing protocol messages sent by the braille display, mapping messages into a form that could be understood by NVDA (key mappings, for example), and getting NVDA to recognize specific input and perform commands and/or send the equivalent command to the operating system. As for braille and HID, what makes it a bit difficult is that the input side is not seen as a typical keyboard - it ic akin to a controller or a specialized input device. If we're dealing with a computer keyboard, then it is better to get the OS/keyboard driver to recognize the input and do whatever it needs to do. But we're talking about an input style that cannot be readily translated into scan codes and virtual key codes (that's how Windows knows about keyboard input), thus the screen reader is involved in:
In other words, the way the term "human interface device" is understood, interpreted, and implemented differs based on who you ask:
As a repository for screen reader development, we fall under the last category, with some folks wearing different hats. Hope all of this makes sense. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
That's what I've been understanding over time as I've been looking into it, that depending on the perspective you approach it from, the focus is different. I see that the available documentation is for developing a driver that understands a braille display, but the perspective from the braille display (firmware) is different. From what you're saying in this message, I understand that not all braille displays send information in the same way. They use the HID protocol but have their own commands. I'm not sure if I understood that part correctly, but if so, then unification still hasn't been achieved because it would still be necessary to make the screen reader able to understand each manufacturer's commands. My proposal is to create a public unified command set that allows for specific commands to be developed for a braille display. This document should be made public so that anyone who wants to develop something with those commands can do so, and to allow the screen reader to understand those commands. This way, we would be developing a standard within the HID protocol that anyone looking for information can find and use. But as long as everyone works separately and creates their own commands, we will never achieve the true goal of the HID braille protocol, which is plug and play functionality. While it may be recognized by the operating system, being recognized by the operating system alone is not enough for it to be functional. It would be good to be able to develop a communication standard with the basic buttons required for a braille display and explained documentation on how to implement it with example code that serves as the foundation for anyone to use those commands to implement it in their development. Something very similar is happening with regular drivers, each manufacturer has its own protocol and commands. So, for each device, an individual driver must be created. In fact, that is the big problem. Hopefully, there would be documentation with a universal driver that anyone could use code or a regular driver with all the documentation so that anyone could replicate the same driver and use it. For example, an open source driver with a recognition standard called "open" for Bluetooth, and any device that has the word "open" at the beginning of the Bluetooth name will be recognized as a device that uses that driver. I think that by implementing it in a screen reader and implementing the first braille display that uses that driver, it would open the door for the driver to be used in the rest of the screen readers. Then, you could truly have the possibility of working with an open source screen reader software and hardware. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
Unless there could be a Standard Windows driver that translates the commands from the braille display into windows standard Input, without having a screen reader. I was hoping USB-IF would have thought about this before shouding out such a standard promising that HID braille displays would work without a screen reader. I think still the example that comes nearest to a Braille input / output interpretation is a dualshock game controler as a HID device. Those have input mapings that need to match some how to keyboard and mouse, and they have output mappings such as vibrations to indicate something which has been sent from the HID host to the HID device.
This could be done via a custom hid miniport driver. Here are all the classes supported so far, but I think Microsoft didn't think of Braille driver API as an own class, so it could be a good start to also talk with them about how to create a custom HID Miniport Driver for Braille Input and Output. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, A completely different direction, but have you looked at BRLTTY and see how it can serve as a bridge between different braille displays and screen readers? Yes, different braille displays implement different protocols, and some have standardized around a common set of messages (protocols) i.e. Baum protocol (talking about braille display protocols will require looking at a history of braille display development for the last twenty years or so). Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
BRLTTY causes many conflicts between screen readers, for example when Narator and braille displays are added via BRLTTY, some of the braille displays are totally overtaken by Narator ynd you cannot find them in NVDA's menu anymore. I had this problem with an user here in Germany and the solution was to remove BRLTTY and all the Braille drivers from Narator. After that the Braille displays appeared again in NVDA's menu and they have been recognized automatically again. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I've also been wanting to do something similar. I have used RP2040 and programmed it using Micropython. Even I have been facing issue in implementing it as Braille HID device. @discapacidad5 It would be great to know if you found out a way on how it's done. I'm very new to this HID thing and it's quiet honestly getting very complicated for me to understand. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I am also trying to make the braille display but I cannot establish communication. |
Beta Was this translation helpful? Give feedback.
-
|
Hi everyone, after a year of working on this code with the help of AI, I believe I've got a good grasp on how it works. However, it's not functioning as expected. I'm hoping someone can help me identify the issue. |
Beta Was this translation helpful? Give feedback.
-
|
HIDBrailleDisplay.h |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @discapacidad5 for sharing your code. I can run it and get it recognised by VoiceOver but nothing else. Are you able to receive Braille Cells Reports? |
Beta Was this translation helpful? Give feedback.
-
|
I have looked at implementing a hid braille display with a RP2040 as well last week since I was tagged in this discussion and I am relatively interested in HID devices in general. I'll see if I can make some time this week to see if I can make a minimal example and put it online. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I am also trying to make the braille display but I cannot establish communication. |
Beta Was this translation helpful? Give feedback.
-
|
Hello guys, I'm working on a new multi-line Braille display project that intends to make use of the "Standard HID Braille Display" option for NVDA. So, the idea is to implement the HID Braille Display usage page so the NVDA software can automatically detect my display and work right away. I'm using an ATSAMD51J20A microcontroller from Microchip and I'm developing the firmware using the Harmony 3 framework for MPLABX IDE. I've created a very simple demo that configures my USB device as a Braille Display with only 1 row of 8 cells, no other feature. However, when I open the NVDA settings, I can't get NVDA to detect my USB Braille Display. By opening the Windows Device Manager tool I can see that my USB device is correctly enumerated. Below is the value of each of the descriptors sent to the USB host. Device DescriptorConfiguration DescriptorInterface DescriptorEndpoint DescritporsReport Descriptor (Here's were the Braille Display usage is defined)Am I configuring something wrong? What could be happening? |
Beta Was this translation helpful? Give feedback.
-
|
I've managed to make some progress on this, but I'm a little bit stuck on interpreting the data being sent to the device. I have some sample code working with and output report being sent every time the screen reader makes an announcement. I get data from the device and it is recognized as a braille display (I had to use the latest version of NVDA). It even automatically starts VoiceOver when it is connected to an iOS device. I am able to receive an unsigned int of length 64, but that only contains a single letter. I don't know if it is sending a stream of data, and I just happen only be reading the first section of it. I don't have a lot of experience with C++ so that it slowing me down a little bit. @blasthenwaytech, I used your report descriptor but added a report id after the application collection. I'm using an ESP32-Wroom-32 device with the NimBLE bluetooth libraries. Below is a screenshot of the output I'm getting. If you convert the data to binary, it corresponds to the braille dots of a single character and the length of the data is 40 (corresponding with the number of cells in the report descriptor) when looking at pCharacteristic.size(). 
I'm thinking it might be serialized data since NVDA is using WriteFile to write the data, but I'm not sure. Here is my forked project from a NimBLE composite HID project: https://github.com/jwlilly/ESP32-BLE-BrailleHID. This is built for the Arduino IDE, but if you follow the instructions in the readme to install the library into the Arduino IDE, you can test it out. Select the "KeyboardLEDs" example. All of my changes can be found in the KeyboardDevice.cpp and KeyboardDescriptors.h files. |
Beta Was this translation helpful? Give feedback.
-
|
Hi there, Thanks again for sharing your progress and the specific issue you're facing with receiving the Braille data on your ESP32 BLE HID project. It's definitely an interesting challenge, and getting it working would be a great step for the community interested in this topic here on GitHub. To help troubleshoot this effectively as a group, understanding the exact state of your code is the most crucial first step. Collaborative problem-solving works best when we can all look at the same code that's producing the behavior. Request for Current Code & Details: Could you please share the current version of the relevant code snippets from your project? Seeing the code that is running right now when you observe the uint64_t (single character) issue would be extremely helpful. This would ideally include: Your actual onWrite callback function (wherever it resides – likely in KeyboardCallbacks or a similar class – handling data received from the screen reader). Any modifications to how the BLE characteristic for the Output report is defined or handled. The essential parts of your main .ino sketch showing initialization. Key setup details (ESP32 board model, library versions like NimBLE, Arduino core version). Analysis and Potential Solution Based on Previously Shared Code: Looking back at the KeyboardDevice.cpp code you shared earlier from your fork, there's a specific point in the KeyboardCallbacks::onWrite function that stands out as a likely cause if your current code is still similar: If your current onWrite function uses getValue<uint64_t>() or getValue<uint8_t>() or any method that reads only a small, fixed number of bytes, it would explain why you're not seeing the full 40 bytes expected from the Braille Output report (which your HID descriptor correctly defines). The uint64_t value would just be the first 8 bytes interpreted as a number. Suggestion to Try: Based on this analysis, if your current onWrite code resembles the above, consider modifying it to read the entire data buffer sent by the host. A standard way to do this with NimBLE is using getLength() and getData(): Moving Forward Together: Please share your current code snippets and setup details. This is the most important step for accurate group diagnosis. Please also try the suggested change in your onWrite function (using getData()/getLength()) if your current code uses a method like getValue<uint64_t>() that reads limited bytes. Crucially, please report back! Share the results of trying the suggested fix. Share the modified code, whether the fix worked or not. If it didn't work, having your current code + the tried modification will give us the best information to figure out the next steps together. Integrating the Solution: Since you're already working within a fork (ESP32-BLE-BrailleHID) based on the ESP32-BLE-CompositeHID library, it would be highly beneficial for everyone if the Braille implementation was structured cleanly. Instead of modifying KeyboardDescriptors.h and KeyboardDevice.cpp directly, consider creating new, dedicated files like BrailleDescriptors.h (containing your Braille HID report descriptor) and BrailleDevice.cpp (containing the specific logic for handling Braille input/output, including the corrected onWrite callback). This approach would cleanly separate the Braille functionality within the composite HID framework. If your changes and fixes work, structuring them this way could bring us much closer to having a definitive, reusable BLE Braille HID solution for the ESP32 that benefits the whole community. That's why sharing the details of what you've done, the specific ESP32 code causing the issue (but which was successfully recognized!), and any subsequent fixes is so important for our collective progress. We're keen to collaborate and crack this BLE Braille HID puzzle. Your detailed feedback and code sharing are key! |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the help! I think we're getting somewhere now. I have updated my repo with some changes. I pull all of my experiments out of the keyboard-related files and into braille-specific files, updated the readme file, and implemented the changes you suggested. I have updated the onWrite function to the following (I had to modify your suggestion a bit since the NimBLE API changed recently, but the change was suggested by the code creator in the migration notes. void BrailleCallbacks::onWrite(NimBLECharacteristic* pCharacteristic, NimBLEConnInfo& connInfo)
{
size_t len = pCharacteristic->getLength(); // Get actual bytes received
ESP_LOGI(LOG_TAG, "Received %d bytes via onWrite", len);
if (len > 0) {
// Get pointer to the raw byte data (uint8_t*)
std::string value = pCharacteristic->getValue();
const uint8_t* data = (uint8_t *) value.data();
if (len == 41) { // Check if we got the expected 41 bytes > 40 cells plus 1 byte for the report id
ESP_LOGI(LOG_TAG, "Processing 41 Braille cell bytes...");
// --- Your Logic Here ---
// Access the 40 bytes via the 'data' pointer
// Example: Print bytes as hex
for (size_t i = 0; i < len && i < len; ++i) {
Serial.printf("%02X", data[i]);
}
Serial.println();
// --- End Your Logic ---
} else {
ESP_LOGW(LOG_TAG, "Received unexpected data length: %d bytes (expected 40)", len);
}
} else {
ESP_LOGW(LOG_TAG, "Received empty data in onWrite");
}
}I am now getting the expected amount of data, but I haven't quite cracked the data conversion. If I convert the hex output to binary, it doesn't seem to correspond to the screen reader announcement. If I do something simple like testing "aaaaaa", that works with an offset of the binary data. Note that I'm only testing with VoiceOver on macOS right now. I haven't tested the changes from today with NVDA. Here is some of the output data: Tested from this simple page: https://codepen.io/jwlilly/full/zxxYYvW |
Beta Was this translation helpful? Give feedback.
-
|
Hi That's fantastic progress! Thank you so much for sharing the update, the code snippet, and the output log. It's really exciting to see you're now receiving the full 41 bytes from the screen reader. Excellent work refactoring the code into dedicated BrailleDevice and BrailleDescriptors files as well – that makes things much cleaner for future integration! Regarding the data interpretation challenge: what you're describing and the hex output you shared look very familiar and promising! I've been working extensively with Braille screen reader data received via serial connections, and I've managed to decode the messages successfully in that context. While HID is a different transport layer, the fundamental data structure sent by the screen reader for the Braille cells should be very similar, if not identical. Your observation about receiving 41 bytes, where the first byte is 0x40 (the Report ID you defined!), is exactly correct. This confirms the HID report structure is working as intended, and the actual Braille cell data (the 40 bytes you care about) starts from the second byte (data[1] in your code snippet). The hex data you posted (40DD00..., 401D00..., etc.) looks very much like the raw cell data I've seen over serial. Regarding the constant stream and interpretation: You mentioned the binary conversion doesn't seem to match the announcement directly yet (except for the simple "aaaaaa" test with an offset). This is also something I've encountered. Here are a few points based on my serial experience that might apply here: Constant Updates: Screen readers often send the entire Braille line (all 40 cells in your case) very frequently, even if only a small part of the screen changes or just the cursor blinks. Cursor Blinking: A very common pattern is seeing the same 40 bytes of cell data sent twice in quick succession, with only one difference: the cursor dots (usually dots 7 and 8) toggling on/off in one specific cell. If you see data streams like: 40 ... [cell data with cursor off] ... 40 ... [cell data with cursor on] ... 40 ... [cell data with cursor off] ... VoiceOver Specifics: VoiceOver (especially on macOS) might have its own quirks in how it represents certain states or characters compared to, say, NVDA. The "offset" you found for "aaaaaa" might be a clue related to this. Decoding might require specific mapping tables or handling rules. How I Can Help: Since I have the decoding logic worked out for serial data, and the raw hex data you're receiving via HID looks so similar, I'm quite confident we can figure out the final interpretation step. The most effective way forward would be if I could run your current code directly on my ESP32 test setup. This would allow me to: See the exact hex output (Serial.printf("%02X", data[i]);) you are getting. Simultaneously see what VoiceOver (or NVDA) is announcing on my end. Apply my existing decoding knowledge directly to the HID data stream you're receiving. This would let me give you much more precise feedback on how to map those bytes (data[1] to data[40]) to the actual Braille dots and characters. We've been trying to achieve reliable BLE Braille HID for a long time, and it feels like you're incredibly close! Would you be willing to ensure your GitHub repo (jwlilly/ESP32-BLE-BrailleHID) is fully up-to-date with the exact code that produced the log output you shared? If so, I can clone it, flash it, and hopefully help crack the final decoding piece together. Thanks again for your work and for sharing the progress! Let's get this solved. |
Beta Was this translation helpful? Give feedback.
-
|
Following up on the idea of creating separate files for the Braille functionality ( However, when I try to compile a basic sketch using this new The error points specifically to the line It seems like this Because of this compilation error, I'm currently unable to build and test the code to try and replicate the original output data issue you were seeing (receiving only partial data/single character) or test potential fixes like adjusting the Could you perhaps take a look at your Getting past this compilation hurdle is the necessary first step so we can properly test the setup and work together on resolving the data interpretation problem. Thanks for your help! |
Beta Was this translation helpful? Give feedback.
-
|
That's a dependency from the original project that I forked. I'm not quite sure what it does, but I figured I would leave the original project alone so we can use it to build a Braille keyboard as well. But the dependency can be installed like this:
And to address your first question, the code in my repo produced the output that I included in my last post, so after you get that dependency installed, it should work. Getting it to work can be a little flaky (at least with my device), I have to do a combination of removing and adding the device from Bluetooth and rebooting the esp32 device before it's recognized as a Braille display by VoiceOver. NVDA might not have these issues if that's how you're testing it. I also occasionally run into an issue after uploading new code to the device where it will not print out my onWrite logs. Unplugging and plugging back in the device usually fixes it |
Beta Was this translation helpful? Give feedback.
-
|
Thanks again for sharing your work on the ESP32-BLE-BrailleHID project and the updates! I've cloned your latest changes and tried running the Braille example provided (examples/BrailleExample/BrailleExample.ino). I made sure to install the "Callback" library dependency as well. The good news is that the code compiles successfully in the Arduino IDE without any errors. However, I'm running into an issue where the ESP32 doesn't seem to be advertising itself over Bluetooth. I've checked using BLE scanner apps on my Android phone and also looked for it in the Bluetooth devices list on my PC, but it doesn't appear in either case. I wanted to check if I might be missing any additional configuration steps beyond just using the example code as-is. Are there any specific board settings, NimBLE configurations (like in menuconfig if not using standard Arduino IDE, although I am), or other prerequisites needed that aren't mentioned in the README? Or could there potentially be something in the current library code preventing the advertising from starting correctly? I just wanted to confirm if I'm using the exact same setup/code as you intended for this example. Any insights you might have would be greatly appreciated! Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
I get a similar issue. It won't pick up on Android, but if you use the nRF Connect app, you can bond it with it. On Windows, iOS, and macOS, I can connect to it, but it doesn't advertise the expected name until it is bonded. On macOS, it is identified as "Bluetooth Device". I think on Windows, it is advertised as just "Input" if I'm not mistaken. Sorry, I forgot to mention that problem. I had the same problem with the project I forked from and haven't dug into that yet. It seems that the name isn't getting added to the advertisement data for some reason. |
Beta Was this translation helpful? Give feedback.
-
|
Have you tried using the keyboard code from the library you saw to see if
it gives the same error? I think it would be good to try the original code
before searching to see if it has the same problem. If you don't have it we
could look for a solution from it. What was the code you forked from?
El mié., 16 abr. 2025 3:07 p. m., John Lilly ***@***.***>
escribió:
… I get a similar issue. It won't pick up on Android, but if you use the nRF
Connect app, you can bond it with it. On Windows, iOS, and macOS, I can
connect to it, but it doesn't advertise the expected name until it is
bonded. On macOS, it is identified as "Bluetooth Device". I think on
Windows, it is advertised as just "Input" if I'm not mistaken.
Sorry, I forgot to mention that problem. I had the same problem with the
project I forked from and haven't dug into that yet. It seems that the name
isn't getting added to the advertisement data for some reason.
—
Reply to this email directly, view it on GitHub
<#14829 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACL5OJVAFC7J5C4ALRAHPVL2Z2TAHAVCNFSM6AAAAABNJPWWNOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBVHA4TAMY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
I believe it had the same issue when I was testing it with the keyboard descriptors just to see if I could get output reports working. I forked from this repo: https://github.com/Mystfit/ESP32-BLE-CompositeHID/blob/ff0b673db2aaef65190e9bd501354db05d8d0b91/BleCompositeHID.cpp#L282 The link above seems to be where the advertising data is being set. I don't see where setName is being called so maybe that is missing. I can look closer at it this evening. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for confirming you were seeing a similar issue with the device not advertising its name correctly! That helped confirm it wasn't just on my end. Based on your confirmation and comparing the It seems the I went ahead and modified the This strongly suggests that the lack of an explicitly advertised name was the main reason for the visibility and connection issues across different platforms. Now, for Android specifically (and possibly others for auto-recognition as Braille), the exact name might be critical. I suspect just "HID" isn't enough for the OS to identify it as a Braille display automatically. I'll need to investigate what the correct expected name should be for Braille devices. However, this current modification should at least solve the fundamental problem of the device being nameless or having a generic name during discovery. Could you please try replacing your I'd be very interested to know if this improves the connection experience on those platforms. Unfortunately, I just realized my main Windows machine doesn't have BLE support, so I can't test it there myself. Your feedback would be invaluable! Here is the modified #include <NimBLEDevice.h>
#include <NimBLEUtils.h>
#include <NimBLEServer.h>
#include "NimBLEHIDDevice.h"
#include "HIDTypes.h"
#include "HIDKeyboardTypes.h"
#include <driver/adc.h>
#include "sdkconfig.h"
#include "BleCompositeHID.h"
#include "BleConnectionStatus.h"
#include <sstream>
#include <iostream>
#include <iomanip>
#if defined(CONFIG_ARDUHAL_ESP_LOG)
#include "esp32-hal-log.h"
#define LOG_TAG "BLECompositeHID"
#else
#include "esp_log.h"
static const char *LOG_TAG = "BLECompositeHID";
#endif
#define SERVICE_UUID_DEVICE_INFORMATION "180A" // Service - Device information
#define CHARACTERISTIC_UUID_SYSTEM_ID "2A23" // Characteristic - System ID 0x2A23
#define CHARACTERISTIC_UUID_MODEL_NUMBER "2A24" // Characteristic - Model Number String - 0x2A24
#define CHARACTERISTIC_UUID_SOFTWARE_REVISION "2A28" // Characteristic - Software Revision String - 0x2A28
#define CHARACTERISTIC_UUID_SERIAL_NUMBER "2A25" // Characteristic - Serial Number String - 0x2A25
#define CHARACTERISTIC_UUID_FIRMWARE_REVISION "2A26" // Characteristic - Firmware Revision String - 0x2A26
#define CHARACTERISTIC_UUID_HARDWARE_REVISION "2A27" // Characteristic - Hardware Revision String - 0x2A27
uint16_t vidSource;
uint16_t vid;
uint16_t pid;
uint16_t guidVersion;
uint16_t hidType;
std::string modelNumber;
std::string softwareRevision;
std::string serialNumber;
std::string firmwareRevision;
std::string hardwareRevision;
std::string systemID;
std::string uint8_to_hex_string(const uint8_t *v, const size_t s) {
std::stringstream ss;
ss << std::hex << std::setfill('0');
for (int i = 0; i < s; i++) {
ss << std::hex << std::setw(2) << static_cast<int>(v[i]);
}
return ss.str();
}
BleCompositeHID::BleCompositeHID(std::string deviceName, std::string deviceManufacturer, uint8_t batteryLevel) : _hid(nullptr), _autoSendTaskHandle(NULL) // Initialize task handle
{
this->deviceName = deviceName.substr(0, CONFIG_BT_NIMBLE_GAP_DEVICE_NAME_MAX_LEN - 1);
this->deviceManufacturer = deviceManufacturer;
this->batteryLevel = batteryLevel;
this->_connectionStatus = new BleConnectionStatus();
}
BleCompositeHID::~BleCompositeHID()
{
// Consider calling end() here to clean up tasks if not done elsewhere
delete this->_connectionStatus;
// if (_hid) { delete _hid; _hid = nullptr; } // NimBLEHIDDevice might be managed by NimBLEServer? Check docs.
}
void BleCompositeHID::begin()
{
this->begin(BLEHostConfiguration());
}
void BleCompositeHID::begin(const BLEHostConfiguration& config)
{
_configuration = config; // we make a copy, so the user can't change actual values midway through operation, without calling the begin function again
modelNumber = _configuration.getModelNumber();
softwareRevision = _configuration.getSoftwareRevision();
serialNumber = _configuration.getSerialNumber();
firmwareRevision = _configuration.getFirmwareRevision();
hardwareRevision = _configuration.getHardwareRevision();
systemID = _configuration.getSystemID();
vidSource = _configuration.getVidSource();
vid = _configuration.getVid();
pid = _configuration.getPid();
guidVersion = _configuration.getGuidVersion();
hidType = _configuration.getHidType();
#ifndef PNPVersionField
// Legacy behaviour for versions of Nimble <= 1.4.1
uint8_t high = highByte(vid);
uint8_t low = lowByte(vid);
vid = low << 8 | high;
high = highByte(pid);
low = lowByte(pid);
pid = low << 8 | high;
high = highByte(guidVersion);
low = lowByte(guidVersion);
guidVersion = low << 8 | high;
#endif
// Start BLE server task
// Increased stack size slightly, adjust if needed
xTaskCreate(this->taskServer, "server", 5120, (void *)this, 5, NULL);
}
void BleCompositeHID::end(void)
{
if(_autoSendTaskHandle != NULL) { // Check if task handle is valid before deleting
vTaskDelete(this->_autoSendTaskHandle);
_autoSendTaskHandle = NULL; // Set handle to NULL after deletion
}
// Optional: Add NimBLEDevice::deinit(true); // true to release memory
ESP_LOGI(LOG_TAG, "BleCompositeHID ended.");
}
void BleCompositeHID::timedSendDeferredReports(void *pvParameter)
{
BleCompositeHID *BleCompositeHIDInstance = (BleCompositeHID *)pvParameter;
ESP_LOGI(LOG_TAG, "timedSendDeferredReports task started.");
if (BleCompositeHIDInstance && BleCompositeHIDInstance->_hid) // Check instance validity
{
std::function<void()> reportFunc;
while(true) { // Loop indefinitely until task is deleted
if(BleCompositeHIDInstance->_deferredReports.ConsumeSync(reportFunc)) { // ConsumeSync waits
if (BleCompositeHIDInstance->isConnected()) { // Only send if connected
reportFunc();
if(BleCompositeHIDInstance->_configuration.getQueueSendRate() > 0) {
vTaskDelay((1000 / BleCompositeHIDInstance->_configuration.getQueueSendRate()) / portTICK_PERIOD_MS);
}
} else {
ESP_LOGD(LOG_TAG, "Deferred report skipped, not connected.");
// Optional: Add a small delay here if not connected to prevent busy-waiting on ConsumeSync if queue fills rapidly?
vTaskDelay(10 / portTICK_PERIOD_MS);
}
} else {
// ConsumeSync returned false, likely because Finish() was called or queue is empty and processing should stop
ESP_LOGI(LOG_TAG, "ConsumeSync returned false, exiting timedSendDeferredReports task.");
break; // Exit loop
}
}
} else {
ESP_LOGE(LOG_TAG, "Invalid instance or HID device in timedSendDeferredReports.");
}
vTaskDelete(NULL); // Delete task when exiting loop
}
void BleCompositeHID::addDevice(BaseCompositeDevice *device)
{
if(device) { // Basic null check
device->_parent = this;
_devices.push_back(device);
} else {
ESP_LOGW(LOG_TAG, "Attempted to add a NULL device.");
}
}
bool BleCompositeHID::isConnected()
{
// Delegate to connection status helper, ensure it's not null
return (this->_connectionStatus != nullptr && this->_connectionStatus->isConnected());
}
void BleCompositeHID::setBatteryLevel(uint8_t level)
{
this->batteryLevel = level;
if (this->_hid)
{
// Notify if connected, otherwise just update internal value
this->_hid->setBatteryLevel(this->batteryLevel, this->isConnected());
}
}
void BleCompositeHID::queueDeviceDeferredReport(std::function<void()> && reportFunc)
{
this->_deferredReports.Produce(std::move(reportFunc)); // Use std::move
}
void BleCompositeHID::sendDeferredReports()
{
if (this->_hid && this->isConnected()) // Check HID and connection status
{
std::function<void()> reportFunc;
while(this->_deferredReports.Consume(reportFunc)){ // Non-blocking consume
reportFunc();
}
}
}
void BleCompositeHID::taskServer(void *pvParameter)
{
BleCompositeHID *BleCompositeHIDInstance = (BleCompositeHID *)pvParameter;
// ESP_LOGI(LOG_TAG, "Setting custom MAC address (example - currently commented out)");
// Use the procedure below to set a custom Bluetooth MAC address
// Compiler adds 0x02 to the last value of board's base MAC address to get the BT MAC address, so take 0x02 away from the value you actually want when setting
// uint8_t newMACAddress[] = {0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF - 0x02};
// esp_base_mac_addr_set(&newMACAddress[0]); // Set new MAC address
ESP_LOGI(LOG_TAG, "Initializing NimBLE device: %s", BleCompositeHIDInstance->deviceName.c_str());
NimBLEDevice::init(BleCompositeHIDInstance->deviceName); // Initialize NimBLE with the device name (for internal use by NimBLE)
ESP_LOGI(LOG_TAG, "Creating NimBLE server...");
NimBLEServer *pServer = NimBLEDevice::createServer();
if (!pServer) {
ESP_LOGE(LOG_TAG, "Failed to create NimBLE server!");
vTaskDelete(NULL);
return;
}
pServer->setCallbacks(BleCompositeHIDInstance->_connectionStatus);
ESP_LOGI(LOG_TAG, "Creating NimBLE HID device...");
BleCompositeHIDInstance->_hid = new NimBLEHIDDevice(pServer);
if (!BleCompositeHIDInstance->_hid) {
ESP_LOGE(LOG_TAG, "Failed to create NimBLEHIDDevice!");
// Clean up server? NimBLEDevice::deinit?
vTaskDelete(NULL);
return;
}
// Setup the HID descriptor buffers
size_t totalBufferSize = 2048; // Should be enough for most composite devices
uint8_t tempHidReportDescriptor[totalBufferSize];
int hidReportDescriptorSize = 0;
ESP_LOGI(LOG_TAG, "Initializing child devices and building HID descriptor...");
// Setup child devices to build the HID report descriptor
for(auto device : BleCompositeHIDInstance->_devices){
if (!device) {
ESP_LOGW(LOG_TAG, "Skipping NULL device pointer in _devices vector.");
continue;
}
const char* currentDeviceName = "Unknown"; // Default name
auto config = device->getDeviceConfig();
if (config) {
currentDeviceName = config->getDeviceName();
}
ESP_LOGD(LOG_TAG, "Before device %s init", currentDeviceName);
device->init(BleCompositeHIDInstance->_hid); // Pass HID device pointer to child
ESP_LOGD(LOG_TAG, "After device %s init", currentDeviceName);
if (!config) {
ESP_LOGE(LOG_TAG, "Device %p has NULL configuration, skipping.", (void*)device);
continue;
}
// Use a temporary buffer for each device's report descriptor part
uint8_t deviceReportBuffer[BLE_ATT_ATTR_MAX_LEN]; // Max size for one part
size_t reportSize = config->makeDeviceReport(deviceReportBuffer, sizeof(deviceReportBuffer));
// Validate the returned size
if(reportSize == 0 || reportSize == (size_t)-1){ // Check for 0 or error (-1 cast to size_t)
ESP_LOGE(LOG_TAG, "Error creating or empty report descriptor for device %s (size: %zu)", currentDeviceName, reportSize); // Use %zu for size_t
continue; // Skip this device if its descriptor is invalid
} else if (reportSize > sizeof(deviceReportBuffer)) {
// This case should technically not happen if makeDeviceReport respects bufferSize
ESP_LOGE(LOG_TAG, "Device %s report size %zu exceeds temporary buffer %zu", currentDeviceName, reportSize, sizeof(deviceReportBuffer)); // Use %zu
continue;
} else {
ESP_LOGD(LOG_TAG, "Created device %s descriptor part with size %zu", currentDeviceName, reportSize); // Use %zu
}
// Check if adding this report part exceeds the total composite descriptor buffer
if (hidReportDescriptorSize + reportSize > totalBufferSize) {
ESP_LOGE(LOG_TAG, "Total HID descriptor size exceeds buffer limit (%zu bytes) while adding device %s. Stopping descriptor build.", totalBufferSize, currentDeviceName); // Use %zu
break; // Stop adding more devices
}
// Append the device's descriptor part to the combined descriptor
memcpy(tempHidReportDescriptor + hidReportDescriptorSize, deviceReportBuffer, reportSize);
hidReportDescriptorSize += reportSize;
}
ESP_LOGI(LOG_TAG, "Final Combined HID Report Descriptor Size: %d bytes", hidReportDescriptorSize); // %d is okay for int
// Set the final combined report map (only if it's valid)
if (hidReportDescriptorSize > 0) {
// Create a correctly sized buffer for the final map
uint8_t finalHidReportDescriptor[hidReportDescriptorSize];
memcpy(finalHidReportDescriptor, tempHidReportDescriptor, hidReportDescriptorSize);
// Log the final descriptor for debugging if needed (use VERBOSE level)
// ESP_LOG_BUFFER_HEXDUMP(LOG_TAG, finalHidReportDescriptor, hidReportDescriptorSize, ESP_LOG_VERBOSE);
BleCompositeHIDInstance->_hid->setReportMap(finalHidReportDescriptor, hidReportDescriptorSize);
ESP_LOGI(LOG_TAG, "HID Report Map set successfully.");
} else {
ESP_LOGE(LOG_TAG, "No valid HID descriptors were added. Cannot set Report Map. Aborting server setup.");
// Clean up resources before exiting task
delete BleCompositeHIDInstance->_hid;
BleCompositeHIDInstance->_hid = nullptr;
// Optional: NimBLEDevice::deinit(true);
vTaskDelete(NULL); // Exit task
return;
}
// Set manufacturer info
BleCompositeHIDInstance->_hid->setManufacturer(BleCompositeHIDInstance->deviceManufacturer);
// Get or Create Device Information Service (0x180A)
// NimBLEHIDDevice likely creates this automatically when setting PnP info,
// but let's get a reference to add more characteristics if needed.
NimBLEService *pService = pServer->getServiceByUUID(SERVICE_UUID_DEVICE_INFORMATION);
if(!pService) {
// If NimBLEHIDDevice didn't create it, we might need to create it manually.
// However, setPnP usually handles this. Log a warning if it's missing.
ESP_LOGW(LOG_TAG, "Device Information Service (0x180A) not found after HID setup. It might be created later by setPnP.");
// If manual creation is needed: pService = pServer->createService(SERVICE_UUID_DEVICE_INFORMATION);
}
// Add Device Information characteristics (Model Number, Serial Number etc.)
// Check if pService exists *after* potentially being created by setPnP below, or ensure setPnP is called first.
// For now, assume setPnP handles service creation and retrieve it *afterwards* if needed.
// Set PnP IDs (Vendor ID, Product ID, Version) - This often creates the 0x180A service if not present
ESP_LOGI(LOG_TAG, "Setting PnP Info: VID=0x%04X, PID=0x%04X, Version=0x%04X, Source=0x%02X", vid, pid, guidVersion, vidSource);
BleCompositeHIDInstance->_hid->setPnp(vidSource, vid, pid, guidVersion);
// Set HID Information (bcdHID=1.11, country=0)
BleCompositeHIDInstance->_hid->setHidInfo(0x00, 0x0111); // HID Version 1.11
// Now retrieve the Device Info service again to add characteristics
pService = pServer->getServiceByUUID(SERVICE_UUID_DEVICE_INFORMATION);
if(pService) {
ESP_LOGI(LOG_TAG, "Setting Device Information Characteristics...");
// Model Number
BLECharacteristic* pChr = pService->getCharacteristic(CHARACTERISTIC_UUID_MODEL_NUMBER);
if(!pChr){ pChr = pService->createCharacteristic(CHARACTERISTIC_UUID_MODEL_NUMBER, NIMBLE_PROPERTY::READ);}
if(pChr) pChr->setValue(modelNumber); else { ESP_LOGE(LOG_TAG, "Failed to create/get Model Number Characteristic"); }
// Software Revision
pChr = pService->getCharacteristic(CHARACTERISTIC_UUID_SOFTWARE_REVISION);
if(!pChr){ pChr = pService->createCharacteristic(CHARACTERISTIC_UUID_SOFTWARE_REVISION, NIMBLE_PROPERTY::READ);}
if(pChr) pChr->setValue(softwareRevision); else { ESP_LOGE(LOG_TAG, "Failed to create/get Software Revision Characteristic"); }
// Serial Number
pChr = pService->getCharacteristic(CHARACTERISTIC_UUID_SERIAL_NUMBER);
if(!pChr){ pChr = pService->createCharacteristic(CHARACTERISTIC_UUID_SERIAL_NUMBER, NIMBLE_PROPERTY::READ);}
if(pChr) pChr->setValue(serialNumber); else { ESP_LOGE(LOG_TAG, "Failed to create/get Serial Number Characteristic"); }
// Firmware Revision
pChr = pService->getCharacteristic(CHARACTERISTIC_UUID_FIRMWARE_REVISION);
if(!pChr){ pChr = pService->createCharacteristic(CHARACTERISTIC_UUID_FIRMWARE_REVISION, NIMBLE_PROPERTY::READ);}
if(pChr) pChr->setValue(firmwareRevision); else { ESP_LOGE(LOG_TAG, "Failed to create/get Firmware Revision Characteristic"); }
// Hardware Revision
pChr = pService->getCharacteristic(CHARACTERISTIC_UUID_HARDWARE_REVISION);
if(!pChr){ pChr = pService->createCharacteristic(CHARACTERISTIC_UUID_HARDWARE_REVISION, NIMBLE_PROPERTY::READ);}
if(pChr) pChr->setValue(hardwareRevision); else { ESP_LOGE(LOG_TAG, "Failed to create/get Hardware Revision Characteristic"); }
// System ID (Example - Make sure systemID string is populated if used)
// pChr = pService->getCharacteristic(CHARACTERISTIC_UUID_SYSTEM_ID);
// if(!pChr){ pChr = pService->createCharacteristic(CHARACTERISTIC_UUID_SYSTEM_ID, NIMBLE_PROPERTY::READ);}
// if(pChr) pChr->setValue(systemID); else { ESP_LOGE(LOG_TAG, "Failed to create/get System ID Characteristic"); }
} else {
ESP_LOGE(LOG_TAG, "Failed to get Device Information Service even after setPnP.");
}
// Set Security Mode
// NimBLEDevice::setSecurityAuth(BLE_SM_PAIR_AUTHREQ_BOND); // Bonding enabled, MITM off, SC off (Legacy Pairing)
// NimBLEDevice::setSecurityAuth(false, false, true); // No Bonding, No MITM, SC enabled
NimBLEDevice::setSecurityAuth(true, false, false); // Bonding enabled, No MITM, SC off (Legacy Pairing - common default)
ESP_LOGI(LOG_TAG, "Security settings configured.");
// Start BLE services (HID, Device Info, Battery)
ESP_LOGI(LOG_TAG, "Starting BLE services...");
BleCompositeHIDInstance->_hid->startServices();
BleCompositeHIDInstance->onStarted(pServer); // User callback
// --- Configure and Start BLE advertisement ---
ESP_LOGI(LOG_TAG, "Configuring BLE advertising...");
NimBLEAdvertising *pAdvertising = NimBLEDevice::getAdvertising(); // Use NimBLEDevice static method is simpler
pAdvertising->setAppearance(hidType); // Set appearance (e.g., HID_GAMEPAD, HID_KEYBOARD, HID_BRAILLE_DISPLAY?)
ESP_LOGI(LOG_TAG, "Advertising Appearance set to: 0x%04X", hidType);
// Advertise the HID Service UUID
if (BleCompositeHIDInstance->_hid && BleCompositeHIDInstance->_hid->getHidService()) { // Check if HID service exists
pAdvertising->addServiceUUID(BleCompositeHIDInstance->_hid->getHidService()->getUUID());
ESP_LOGI(LOG_TAG, "HID Service UUID added to advertisement.");
} else {
ESP_LOGE(LOG_TAG, "HID Service is NULL, cannot add UUID to advertisement!");
}
// --- MODIFICACIÓN CLAVE INICIO ---
// Explicitly set the advertised device name
pAdvertising->setName(BleCompositeHIDInstance->deviceName);
ESP_LOGI(LOG_TAG, "Advertised Name set to: %s", BleCompositeHIDInstance->deviceName.c_str());
// Enable Scan Response (recommended for better discovery and sending full name if needed)
pAdvertising->enableScanResponse(true);
ESP_LOGI(LOG_TAG, "Scan Response enabled.");
// Optional: Set specific Scan Response data if needed
// pAdvertising->setScanResponseData(...);
// --- MODIFICACIÓN CLAVE FIN ---
// Start advertising
ESP_LOGI(LOG_TAG, "Starting advertising...");
bool success = pAdvertising->start();
ESP_LOGI(LOG_TAG, "pAdvertising->start() call returned: %s", success ? "true" : "false");
if(success) {
ESP_LOGI(LOG_TAG, "Advertising started successfully as '%s'", BleCompositeHIDInstance->deviceName.c_str()); // Use INFO for successful start
} else {
ESP_LOGE(LOG_TAG, "Advertising failed to start!");
// Consider adding error handling here
}
// Update battery level (initial level)
BleCompositeHIDInstance->setBatteryLevel(BleCompositeHIDInstance->batteryLevel);
ESP_LOGI(LOG_TAG, "Initial battery level set to: %d", BleCompositeHIDInstance->batteryLevel);
// Start timed auto send for deferred reports (if enabled in config)
if(BleCompositeHIDInstance->_configuration.getQueuedSending()){
ESP_LOGI(LOG_TAG, "Starting timedSendDeferredReports task...");
// Increased stack size slightly, adjust if needed. Priority 5 is standard.
xTaskCreate(BleCompositeHIDInstance->timedSendDeferredReports, "autoSend", 4096, (void *)BleCompositeHIDInstance, 5, &BleCompositeHIDInstance->_autoSendTaskHandle);
if (BleCompositeHIDInstance->_autoSendTaskHandle == NULL) {
ESP_LOGE(LOG_TAG, "Failed to create timedSendDeferredReports task!");
}
} else {
ESP_LOGI(LOG_TAG, "Queued sending task not enabled.");
}
// Task setup is complete, NimBLE handles events in its own tasks.
ESP_LOGI(LOG_TAG, "taskServer setup finished. Task will exit.");
vTaskDelete(NULL); // Allow this setup task to complete and be removed
} |
Beta Was this translation helpful? Give feedback.
-
|
I'll give that a shot! Thank you! In the mean time, this is what I got "HID" from for the device name: https://github.com/google/talkback/blob/26a27dc009d5b3605e744222541f045a3c24e038/braille/brltty/src/phone/java/com/google/android/accessibility/braille/brltty/SupportedDevicesHelper.java#L539 It seems that Android is filtering known devices and applying the communication methods using what I guess is the name of the device. But there may be more to it. Do you have access to a Braille HID device that actually connects to Android using HID? We could attempt to emulate the device data if so. And now that I'm looking at the TalkBack source a little closer, it specifies this in a comment above the line I linked: // HID devices
// The keys match the key set in
// //depot/google3/third_party/brltty/Drivers/Braille/HID/brldefs-hid.h;rcl=626237007;l=16`I'm not sure what brldefs-hid.h contains, but if that file is public, it might help us figure out the Android piece. |
Beta Was this translation helpful? Give feedback.
-
|
I was looking for drivers on nbda for HID and they don't really take the
name into account and it's very likely that on Android they won't either
because they look for devices connected by USB in principle. I think it
has more to do with the name of the manufacturer and another code related
to the developer and I think it would be an alternative to get a braille
display that connects via HB pair.
I don't have one anymore, I would have solved that problem a long time ago
because with a braille display you could understand how it works. That's
why we've been working on trial and error tests all this time, but we're
going to get there. I still haven't gotten him to send me any
texts.However, you have managed to get me to answer something on Windows
and on iPhone. I don't have an iPhone, but I will try to see how I can get
some answers on Windows.
El mié., 16 abr. 2025 4:54 p. m., John Lilly ***@***.***>
escribió:
… I'll give that a shot! Thank you! In the mean time, this is what I got
"HID" from for the device name:
https://github.com/google/talkback/blob/26a27dc009d5b3605e744222541f045a3c24e038/braille/brltty/src/phone/java/com/google/android/accessibility/braille/brltty/SupportedDevicesHelper.java#L539
It seems that Android is filtering known devices and applying the
communication methods using what I guess is the name of the device. But
there may be more to it. Do you have access to a Braille HID device that
actually connects to Android using HID? We could attempt to emulate the
device data if so. And now that I'm looking at the TalkBack source a little
closer, it specifies this in a comment above the line I linked:
// HID devices
// The keys match the key set in
// //depot/google3/third_party/brltty/Drivers/Braille/HID/brldefs-hid.h;rcl=626237007;l=16`
I'm not sure what brldefs-hid.h contains, but if that file is public, it
might help us figure out the Android piece.
—
Reply to this email directly, view it on GitHub
<#14829 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACL5OJVN6HDFNSLK2TGPU532Z27SHAVCNFSM6AAAAABNJPWWNOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBVHE3DONA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for creating the issue on the NimBLE-Arduino repo (h2zero/NimBLE-Arduino#936)! That's a great step, and hopefully, someone familiar with NimBLE's internals, especially regarding GATT server state after bonding, might have some insights. Building on that idea, and thinking about where the connection failure specifically manifests (seemingly only on Android via TalkBack, after bonding, despite correct advertising), it might also be beneficial to reach out to the Brltty project and perhaps the official TalkBack repository as well. My reasoning is: TalkBack's Reliance on Brltty: As you pointed out earlier, TalkBack heavily relies on Brltty for its underlying Braille device drivers and connection logic. HID Implementation Specifics: While NimBLE provides the BLE transport, the actual interpretation and handling of the HID Braille standard on Android happen within Brltty and TalkBack. They might have specific expectations or encountered similar quirks regarding HID service discovery or state management after a device is bonded. Targeted Expertise: Developers working specifically on TalkBack's/Brltty's Braille HID integration might have encountered similar post-bonding issues or know specific Android requirements or workarounds that aren't obvious from the BLE library perspective alone. Perhaps cross-posting the core issue (services seemingly disappearing from the GATT server only after bonding on Android, preventing connection) to the Brltty mailing list/GitHub or the TalkBack issue tracker could potentially shed some light from their perspective or catch the eye of a developer familiar with this exact scenario. Just a thought, as getting input from the 'consuming' side might help us get unstuck on this tricky post-bonding problem. Thanks again for driving this forward! |
Beta Was this translation helpful? Give feedback.
-
Hi @jwlilly, |
Beta Was this translation helpful? Give feedback.
-
|
Sure, I can look into posting on the issue trackers for TalkBack and Brltty this week. I did some experiments over the weekend on this. It doesn't help us very much right now, but I have a rooted Android device. I also have some experience with Frida (a security hacking tool), so I wrote a script to override the |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jwlilly, I was digging deeper into the TalkBack source code you shared earlier, specifically looking at how connections are handled, and stumbled upon something potentially very significant in At the top of that file, there's a prominent annotation: @RequiresApi(api = 35)
public class BtHidConnector extends HidConnector {
// ... class implementation ...
}This API level 35 corresponds to Android 15, which is the current stable version available on many devices. Implication: If this interpretation is correct, it would mean that on Android versions prior to 15 (like Android 14, 13, 12, 11, etc.), TalkBack would not even attempt to use the My Testing Context: This aligns with my experience, as all my tests have been conducted on a device running Android 11. If the BLE HID support for Braille was indeed introduced or properly enabled only starting with Android 15, that would explain why it fails for me and potentially for others on older Android versions. Request for Testing: This leads to a crucial question: Do you happen to have access to a device running Android 15 (API 35)? If you do, could you possibly test the current ESP32 code (with the corrected advertising including both 0x1812 and 0x1124 UUIDs) on that Android 15 device? It would be incredibly valuable to know if TalkBack successfully connects using HID on API 35. If it does connect on Android 15 but not on earlier versions, it would confirm that the limitation isn't in our ESP32 code or the NimBLE library, but rather in the minimum Android version required by TalkBack for BLE HID Braille support. This could save us a lot of time trying to debug the ESP32 side if the actual requirement lies with the Android OS version. Let me know if testing on Android 15 is feasible for you! Thanks again for all the collaboration. |
Beta Was this translation helpful? Give feedback.
-
|
I just created an issue for TalkBack requesting information on bluetooth HID devices: https://issuetracker.google.com/issues/414491377 I also forgot to respond to the Android 15 question. I do have Android 15 and TalkBack 15.2, so my device and TalkBack should support the HID protocol. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello,
I hope this message finds you well. I am reaching out to you because I am working on an open-source project called BrailleTouch. Our goal is to create affordable Braille displays for blind and deaf-blind individuals.
We are currently stuck and unable to get the HID Braille to receive information from the screen reader and continue programming. As someone with experience in this area, your expertise could be of great help to our project.
I believe that an open-source Braille display could be incredibly useful and have many benefits for those who are blind and deaf-blind. Your knowledge of the HID protocol could be particularly helpful in implementing the protocol in our Braille display.
Please take a moment to review our project on GitHub at https://github.com/brailletouch. If you're interested in helping, please let me know.
Thank you for your time and consideration. I look forward to hearing back from you.
Best regards,
Cergio Monasterio
Beta Was this translation helpful? Give feedback.
All reactions