CSV 特殊字符的处理

在 CSV 文件中，如果一个 field 的字符包含回车等特殊字符，会怎么样呢？通过一个 golang 程序验证一下。

不同字符的写操作

针对各种不同的字符类型，造一些模拟数据写入 csv 文件。

var bs = string([]byte{0x08})
var return0D0A = string([]byte{0x0D, 0x0A})

var lines = [][]string{
	{
		"normal",
		"this-is-normal-1",
		"this-is-normal-2",
		"this-is-normal-3",
	},
	{
		"space",
		"this is space 1",
		"this is space 2",
		"this is space 3",
	},
	{
		"enter-n",
		"this is enter 1\n",
		"this is\n enter 2",
		"\nthis is enter 3",
	},
	{
		"enter-r",
		"this is enter 1\r",
		"this is\r enter 2",
		"\rthis is enter 3",
	},
	{
		"enter-r-n",
		"this is enter 1\r\n",
		"this is\r\n enter 2",
		"\r\nthis is enter 3",
	},
	{
		"backspace",
		"this is bs 1" + bs,
		"this is" + bs + " bs 2",
		bs + "this is bs 3",
	},
	{
		"return-0d0a",
		"this is 0d0a 1" + return0D0A,
		"this is" + return0D0A + " 0d0a 2",
		return0D0A + "this is 0d0a 3",
	},
}

写入完成后，检查输出文件：

直接打开文件，有预期输出。特殊字符会使用引号处理。

normal,this-is-normal-1,this-is-normal-2,this-is-normal-3
space,this is space 1,this is space 2,this is space 3
enter-n,"this is enter 1
","this is
 enter 2","
this is enter 3"
this is enter 3" enter 1
enter-r-n,"this is enter 1
","this is
 enter 2","

通过 golang 再读取数据，跟原始数据对比，结果出现了错误。enter-r-n和return-0d0a两个场景下，csv 读出来和原始数据不符。

exp
[
this is 0d0a 3]
0d0a7468697320697320306430612033
act
[
this is 0d0a 3]
0a7468697320697320306430612033

可以看到字符0d0a中的0d被删除了。

查看写出来的原始文件

	f2, _ := os.Open(path)
	defer f2.Close()
	contents, _ := ioutil.ReadAll(f2)
	fmt.Println(hex.EncodeToString(contents))

发现数据中的0d0a是正确的。怀疑有 2 个地方可能导致0d的丢失：

读文件过程
[]byte 类型转换为 string 类型

读文件过程不容易检查，另外尝试类型转换是否会造成0d丢失。

func main() {

	r := []byte{0x0d, 0x0a}

	rs := string(r)
	fmt.Printf("rs length %d \n", len(rs))

	rsBack := []byte(rs)

	fmt.Printf("r ori: %s \n", hex.EncodeToString(r))
	fmt.Printf("r back: %s \n", hex.EncodeToString(rsBack))
}

输出为

rs length 2
r ori: 0d0a
r back: 0d0a

证明在数据类型转换过程中，并不会导致0d丢失。

因此判断，golang csv 在读取文件时，会把0d0a替换为0a