[ResponseAPI] Further polish message serialization and unit tests

[Jialin]

Purpose

Address leftover comments in #26620

• hasattr checks
• unit test location

Test Plan & Test Result

With the change, unit tests are still finished successfully

> pytest tests/entrypoints/openai/test_protocol.py 
> pytest tests/entrypoints/openai/test_response_api_with_harmony.py -k test_streaming

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[Jialin]

CC @qandrew @yeqcharlotte

[gemini-code-assist]

Code Review

This pull request refactors the unit tests for message serialization and improves the serialization logic. The movement of tests into a dedicated file test_protocol.py is a good change for code organization. The update in serialize_message to check for hasattr(msg, "to_dict") is more explicit and robust than checking for __dict__.

However, I've found a critical issue in the serialize_message function where it can return inconsistent types (a dict or a str), which will lead to incorrect JSON serialization for certain messages. I've left a detailed comment with a recommended fix. Addressing this is crucial for the correctness of the Response API.

[yeqcharlotte]

thanks for the change!